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Finding Fortune in Thirteen Out-of-School Time Programs 

Introduction 



Overview 

Finding Fortune in Thirteen Out-of-School Time 
Programs builds on the American Youth Policy 
Forum’s (AYPF) previous efforts to provide youth 
policymakers, educators, and practitioners easy-to- 
read, accessible, at-your-finger-tip guides to research- 
proven, effective practices affecting youth, and to 
disseminate these findings in useful compendia 
formats. Five of AYPF’s compendia of evaluations of 
effective youth programs are entitled as follows: 1) 
Some Things DO Make a Difference for Youth , 2) 
MORE Things That DO Make a Difference for Youth , 
3) Raising Academic Achievement, 4) Raising 
Minority Academic Achievement, and 5) No More 
Islands: Family Involvement in 27 School and Youth 
Programs. These compendia serve as useful references 
that provide succinct program information about 
youth populations, and evidence of effectiveness 
and contact information. 

The present compendium, which summarizes the 
evaluations of 13 Out-of-School Time (OST) pro- 
grams, represents AYPF’s continued commitment to 
establish “what works” and identify “best practices” 
in programs that serve young people. OST refers to 
“programs, activities, and opportunities during non- 
school hours where staff are engaged in promoting 
the overall development of school-aged children and 
youth ages 6-18” (NIOST, 2003). The term OST 
represents a shift from “after-school,” which often 
referred to programs that were narrowly focused on 
only providing academic assistance and a safe place 
for children in the non-school hours. However, the 
advent of the youth development movement forced 
programs to move beyond keeping youth safe and 
academically competent to providing youth with a 
sense of belonging, leadership skills, opportunities 
for input, and decision-making ability in programs, 
and challenging and interesting activities (Gambone 
& Axbreton, 1997). In short, OST programs, unlike 
after-school programs, involve efforts to comprehen- 
sively and holistically serve young people. 



AYPF believes that it is critical to draw attention to 
OST programs for a number of reasons. First, youth 
participation in OST programs has been linked to 
increased academic, developmental, and social out- 
comes for young people. Next, youth who participate 
in OST programs have shown positive outcomes in 
school attachment, academic achievement, supportive 
relationships with adults, and peer relationships. 

Lastly, OST programs offer youth opportunities to 
engage in cultural activities, multiple intelligence 
learning environments (e.g., an OST program may 
use music to teach writing rather than solely using 
direct instruction), and increase their participation 
in artistic and recreational activities. 

Research has shown that participation in creative and 
recreational activities can be associated with higher 
self-concept, self-esteem, and resiliency in youth. 

Due to decreased funding in the arts, music, and 
recreation, schools may no longer provide these 
activities. However, OST programs offer opportunities 
to compensate for this loss. 

Further, the changing demographics of society 
from majority two-parent families to the increasing 
number of single-parent families highlights the 
importance of OST programs for providing safe, 
structured, and engaging activities for youth in the 
hours after school. The critical hours from 3:00 pm 

Percent of Violent Juvenile Crime Occuring 
Each Hour on School Days 
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to 6:00 pm, when most parents are working, are 
prime time for youth participation in criminal activity 
or as victims of violence. Therefore, participation in 
OST programs during this time may reduce the 
number of youth victims of crime and delinquency. 

In reviewing the research literature on OST programs, 
AYPF found conflicting research linking youth out- 
comes to program participation. Some evaluations 
of OST programs have shown that youth who attend 
programs have higher academic grades, school 
engagement, and prosocial behavior than their peers 
that do not attend programs. Other research has 
shown no difference between students who attend 
programs and their peers who do not attend programs. 
Because research has demonstrated contrasting out- 
comes for young people, the ability to make the case 
that program participation is associated with positive 
youth outcomes is constrained (an expanded discus- 
sion on this subject will follow in the Key Issues in 
Research section of this report). 

Only through continuous, high quality, and rigorous 
evaluations, and the establishment of best practices 
in the field will consistency of outcomes be 
established in the OST field. 



The evaluations of the 13 OST programs that do 
show positive outcomes for young people, will: 

• Contribute to a better understanding of how pro- 
grams increase youth outcomes and identify key 
factors or components of effective programs; 

• Provoke meaningful conversations on how to 
develop and sustain model programs; and 

• Help policymakers, practitioners, and advocates 
systematically understand the overall picture of 
OST programming. 

Thirteen may be an unlucky number, but we hope 
that these 13 programs will help policymakers, prac- 
titioners, and others “find fortune” in the academic 
and developmental outcomes in OST programs. 

This report includes four parts: 

• An introduction to out-of-school time program- 
ming and to the findings of this report. 

• A discussion of key issues related to research on 
out-of-school time programs. 

• An analysis of the evaluations including 
outcomes, program components and contributing 
factors. 

• Recommendations to the youth policy field 
regarding out-of-school time programming. 
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Key Issues in OST Research 



This out-of-school time (OST) compendium summa- 
rizes and reviews the evaluations of 13 OST programs 
with positive outcomes for young people. A careful 
review of the evaluation summaries showed that 
many evaluations could have produced even more 
useful results if there was more focus in the following 
areas: (1) improvement in the reliability of research 
designs; (2) additional information on survey 
instruments; (3) examination of both academic 
and developmental outcomes; and (4) disaggregation 
of the research data. 

One factor that makes research designs less reliable 
is selection bias. Since most program participation is 
voluntary, students who participate in programs are 
usually highly-motivated or have highly-motivated 
parents, so they may not represent the “average” 
student (Olsen, 2000). If selection bias occurs, 
drawing specific conclusions about the link between 
program interventions and positive youth outcomes 
becomes difficult. In other words, programs may 
show better outcomes for program students, but only 
when those students are compared to their non- 
program peers who are less motivated. 

When research relies solely on data from students 
participating in the program, it may not be as reliable 
as other objective data that could be collected. The 
best way to control for selection bias and many other 
problems is through the use of adequate control 
groups. The use of control groups allows the differ- 
ences between the program group and the control 
group (non-program students) to be attributable to 
the program intervention (Rossi & Freeman, 1993). 

Another problem can arise if low numbers of 
program students participate in evaluations. If that 
happens, a complete picture cannot be drawn from 
program evaluations. There are a variety of different 
reasons that students do not participate in evaluations: 

(1) unsigned parental consent forms; 

(2) absence on days that evaluation occurs; and 

(3) incomplete student data in program files. 



Research designs can be made more rigorous if: 

• random assignment is implemented. Usually, 
program directors disagree with the technique of 
random assignment, because students who need 
services will not be able to participate in the 
program. However, non-program students could 
participate in the program at a later date. 

• matched constructed controls are utilized. 
Students who receive the program intervention 
are matched with an equivalent group, not 
selected randomly, from whom the intervention 
is withheld. 

• repeated-measures reflexive controls are 
employed. This allows for the same students 
to be observed repeatedly over time (Rossi & 
Freeman, 1993). 

• evaluators communicate more clearly with 
program sites as to what information is needed 
to secure complete student program files. 

• technical assistance is offered, by intermediary 
or funding agencies, to help programs better 
manage student files. 

Gottfredson, Gottfredson and Weisman (2001) argue 
that weak research designs allow both advocates and 
critics of programs to use the same research to 
support their position on OST programs. For 
example, OST advocates use findings from the 
Quantum Opportunities Program (QOP) to show that 
positive youth outcomes can be linked to program 
participation (see evaluation summary for more 
information about QOP). Findings from the QOP 
research reveal that participants were more likely to 
be high school graduates, attend postsecondary 
schools, and less likely to be teen parents or to be 
arrested than their non-program peers. Despite the 
positive findings, opponents of OST funding point 
out shortcomings in the program’s research design, 
such as the small evaluation sample and the select 
use of program sites included in the evaluation 
(Olsen, 2000). “For OST opponents, the glass con- 
taining evidence that program outcomes can be 
linked to participation is half-empty” (Gottfredson, 
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Gottffedson, &Weisman, 2001, p.81). Therefore, 
providing rigor to evaluation research is crucial 
in producing useful and credible results in OST 
research (Rossi & Freeman, 1993). 

Another factor that would contribute to the continuous 
development of evaluation research is the provision 
of additional information on survey instruments as 
part of evaluation reports. Evaluations often use 
survey data to identify outcomes linked to program 
participation. However, reports often provide little 
information on the survey instruments themselves. 
Additional information might include copies of the 
survey instruments and, identification of the items 
within subscales and reliability scores. Further, 
information on the population on which the survey 
was normed should be provided. 

Seligson cautions that program outcomes “should be 
broad so that one does not look only at test scores, 
but also the social and emotional development of 
the child (Hart, 2000).” She states, further, “in the 
current policy climate, we are assuming these pro- 
grams can make a huge positive impact on children’s 
achievement, and therefore narrowing the agenda to 
academic learning rather than care. OST programs 
are about both care and learning.” In short, evalua- 
tions need to tell both sides of the developmental 
story of adolescents, by examining not only the 
effects of participation on academic outcomes but 
also on how programs may affect adolescents’ 
resiliency, self-concept, school engagement and 
other vital areas of developmental opportunity. 

Erikson’s (1968) psychosocial developmental theory 
describes how an adolescent’s search for identity can 
be manifested through participation in OST activities. 
When adolescents participate in OST activities, they 
have an opportunity to develop their sense of self. 
“Youth seek their true selves through peer groups, 
clubs, sports, religious, and other activities. These 
groups provide opportunities to try out new roles 
much in the way someone might try on jackets in a 



store until finding one that fits” (Miller, 1983, p. 
166). A youth’s full development into adulthood 
involves more than just what can be measured by 
academic outcomes alone. 

The importance of the disaggregation of research 
data is another key issue, raised from the 21 st 
Century Community Learning Centers (CCLC) 
study findings reviewed in this report. Although 
there were no differences between program students 
and non-program students on academic outcomes, 
the disaggregation of the research data found that 
African American students showed increased effort 
in the classroom, reduced lateness for school, and 
increased math grades as a result of their program 
participation. Impacts on math grades and promptness 
were also evident for Latino students. None of these 
impacts were evident for white students. 

There is growing evidence that OST programs have 
increased achievement gains for Latino and African 
American students. The LA’s BEST (Better Educated 
Students for Tomorrow) program helped Latino 
students become more proficient in English than 
their peers who did not attend the OST program. In 
this same context, the Sacramento START (Students 
Today Achieving Results for Tomorrow) program 
showed greater gains for English Language Learner 
students compared to English speakers on math out- 
comes. Such findings from OST research suggest 
that programs may be beneficial in lowering the 
achievement gap for minority students. Therefore, 
more disaggregation of research data is needed. 

By continuing to develop and improve evaluation 
research, the OST field may be more accurately tai- 
lored to student needs, which will result in services 
that improve students’ lives, participation in school, 
and close the gap between minority students and 
their white peers. Improving the quality of evaluation 
research would help improve the quality of services 
for children and youth. 
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Report Methodology 



This report is the sixth in a collection of American 
Youth Policy Forum (AYPF) compendia of evaluations 
of effective youth programs. The five other compendia 
are: Some Things DO Make a Difference for Youth: 

A Compendium of Evaluations of Youth Programs 
and Practices (1997); MORE Things That DO Make 
a Difference for Youth (1999); Raising Academic 
Achievement (2000); Raising Minority Academic 
Achievement (2001); and No More Islands: Family 
Involvement in 27 School and Youth Programs 
(2003). (The methodology used for these AYPF 
compendia is included as Appendix I.) 

These compendia of evaluations of effective programs 
represent a broad range of youth-related initiatives, 
including early childhood through postsecondary 
education, English language development, career 
preparation, employment and training, service-learn- 
ing, voluntary youth serving organizations, rehabilita- 
tion programs for youth offenders, and teenage preg- 
nancy prevention programs. Programs were included 
in the earlier compendia because of their documented 
results for young people such as improved school 
attendance, classroom and home behavior; higher 
grade point averages; lower grade retention; lower 
demand for special education services; increased 
college attendance and lower participation in risk- 
taking activities and consequences (such as dropping 
out of school, alcohol use, pregnancy or court 
involvement). 

AYPF set out to find evaluations of out-of-school 
time programming that met our basic criteria for 
inclusion in the AYPF compendia. These criteria 
included: 

(1) program characteristics — programs and prac- 
tices had to target school-aged children and, 
ideally, adolescents; 

(2) research quality — evaluation sample, design, 
and methodology had to follow accepted 
research standards; and 

(3) program results/outcomes — the evaluations had 
to include quantitative data indicating the initia- 



tive resulted in positive effects on participants, 
such as improved academic achievement, 
increased graduation rates, decreased rate 
of risky behaviors, and others. 

The search for evaluations included: 

(1) summaries of updated and previous out-of- 
school time program evaluations published in 
prior AYPF compendia; 

(2) reviews of national databases, such as the 
Educational Resources Information Center 
(ERIC), Sociological Abstracts, Harvard Family 
Research Program (HFRP), The After-School 
Corporation (TASC), Promising Practices 
Network (PPN), National Governors Association 
(NGA) and Center for Research on the 
Education of Students Placed at Risk 
(CRESPAR); 

(3) internet searches; and 

(4) direct contact with program coordinators, 
policymakers, funding officers and researchers, 
including staff of the Harvard Family Research 
Project and the Academy for Educational 
Development. 

Six of the out-of-school time program evaluation 
summaries in this report were published in one of 
the other AYPF compendia: 4-H, Beacons, Big 
Brothers Big Sisters, Boys and Girls Clubs of 
America, Quantum Opportunity Program, and 
Sacramento START. The other evaluation summaries 
are new: 21st Century Community Learning 
Centers, BELL After-School Instructional 
Curriculum, Cap City Kids Program, Juvenile 
Mentoring Program, LA’s Best, The After-School 
Corporation (TASC) and Youth Education for 
Tomorrow (YET). 

Each evaluation summary includes the following 
sections: 

• Overview provides a brief look at how the 
program started and its purpose. 

• Population offers demographic information on 
the population served. 
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• Key Findings summarizes the impact of the 
program on the target population. Data on 
program costs, cost/benefit data, and levels of 
statistical significance of findings are provided 
whenever available. 

• Program Components describes the basic 
elements or structure of the program. 

• Contributing Factors summarizes the factors 
that most strongly contributed to the program’s 
positive results, either according to information 
directly supplied by the evaluators or our 
understanding of the evaluations, confirmed 
by the evaluators. 

• Study Methodology briefly describes the design 
of the evaluation. 

• Evaluation Funding indicates what entity or 
entities funded the evaluation. This provides the 
reader some understanding of the objectivity of 
the evaluator. 



• Geographic Areas identifies the site of the 
programs evaluated. 

• Contact Information is provided for both the 
evaluators and program implementers. 

While the brevity of the summaries facilitates 
reading, it limits the information that can be provided. 
Readers are strongly encouraged to consult the 
original evaluations cited at the beginning of each 
summary and to contact the evaluator or program 
staff for more information , especially regarding 
study methodology . 

After summarizing the program evaluations, AYPF 
analyzed the results in three ways: 

• What are common youth outcomes? 

• What are common program components? 

• Which contributing factors seemed to lead to 
positive outcomes? 



Note: The evaluation summaries of the 4H and BB/BS programs are reprints of summaries that were published in previous compendia. As reprinted, 
the summaries did not indicate whether differences between program and non-program students or pre- and post measurements were statistically 
significant. However, evaluation summaries developed for the present compendium note whether differences between measurements or program and 
non-program student outcomes are statistically significant. 
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The Compendium Programs 



• 4-H — nationwide — is the largest voluntary 
co-educational program in the world. Evaluation 
focused on a special 4-H after-school program 
implemented in public housing in Kansas City, 
MO. The program was primarily for 5- to 11- 
year olds, with teenage public housing residents 
serving as mentors. 

• 21 st Century Community Learning Centers — 

nationwide — In 1994, the 21 st Century 
Community Learning Center (21 st CCLC) 
program was authorized by Congress to open 
schools for broader use by communities, including 
the creation of after-school programs. These pro- 
grams include tutorial services, academic enrich- 
ment activities to help students meet local and 
state academic standards in reading and math, 
and youth development activities including: drug 
and violence prevention programs, technology 
education programs, music, and recreation. 

• Beacons — San Francisco and New York — are 
community centers located in public school 
buildings that offer a range of services for par- 
ticipants of all ages, such as educational enrich- 
ment, after-school child-care, recreation, voter 
registration, cultural events, immigrant supports, 
health and mental health referrals, and substance 
abuse and pregnancy prevention. This study 
focuses on Beacon centers in New York City. 

• BELL After-School Instructional 
Curriculum — Boston, New York and Washington, 
DC — In 1992, two members of the Harvard 
Black Law Student Association founded Building 
Educated Leaders for Life (BELL), a commu- 
nity-based, organization. The BASICS (BELL 
After-School Instructional Curriculum) program 
run by BELL meets for three hours after every 
school day, for thirty weeks, offering students 
concentrated small group tutoring and mentoring. 

• Big Brothers, Big Sisters (BB/BS) — 

nationwide — is a 93 -year-old program whose 
autonomously funded local affiliates support 
one-to-one mentoring matches between 
volunteer adults and young people. 

• Boys and Girls Clubs of America (B&GCA) — 

nationwide — founded in 1906 and has more than 



2,000 facilities in all 50 states, Puerto Rico, 

U.S. Virgin Islands and U.S. military installations 
abroad. Nationally, more than 2,000 clubs serve 
over three million youth. In 1996, B&GCA 
piloted an after school educational enhancement 
program for youth in public housing in five 
cities. This evaluation looks at the results of 
the pilot study. 

• Cap City Kids Program (CCK) — Columbus, 
OH — The Cap City Kids (CCK) after-school 
program was initiated by the Mayor’s office, 

to improve youth educational outcomes and the 
access to positive youth development programs. 
It was piloted in four community recreation 
centers located in disadvantaged neighborhoods. 

• Juvenile Mentoring Program — nationwide — 
The Juvenile Mentoring Program (JUMP) was 
implemented in 1995, by the U.S. Off ice of 
Juvenile Justice and Delinquency Prevention to 
serve youth at-risk for delinquency, gang 
involvement, educational failure, and dropping 
out-of-school by beginning or continuing one-to- 
one mentoring relationships between adult vol- 
unteers and at-risk youth. To date, there are over 
260 agencies in 48 states/U.S. territories that 
have received JUMP support. 

• LA’s Best (Better Educated Students for 
Tomorrow) — Los Angeles, CA — an after school 
program for students in grades K-5, including a 
safe environment, educational enrichment pro- 
grams, and recreational activities primarily for 
minority students and youth from low-income 
families. Daily enrichment activities include 
homework assistance, computer activities, drama, 
sports, visual arts, reading, and field trips. 

• Quantum Opportunities Project (QOP) — 

Philadelphia, PA; Saginaw, MI; Oklahoma City, 
OK; San Antonio, TX; and Milwaukee, WI — a 
year-round, multi-year, comprehensive service 
program for disadvantaged youth (families 
receiving food stamps and public assistance) 
launched in five communities in 1989. The QOP 
program provided each site of 25 youth with a 
caring adult that served as both role model and 
advisor to each student for four years. 
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Sacramento START — Sacramento, CA — 

An after-school program providing a safe, edu- 
cational learning environment for elementary 
school students from low-income families. 

The After-School Corporation (TASC) — 

New York — TASC’s mission is to start or expand 
school-based after-school programs that are oper- 
ated by non-profit or community-based organiza- 
tions (CBO). Through several funders, TASC has 
funded more than 130 CBOs and other non-profit 
organizations in 143 New York City public 
schools and 75 schools in New York State. 



• Youth Education for Tomorrow (YET) — 

Philadelphia, PA — established by Public/Private 
Ventures (PPV) to explore the capacity and 
potential of faith-based institutions in three 
areas: literacy, mentoring, and child-care services. 
This evaluation focuses on literacy. Sites were 
located in schools, community-based organiza- 
tions, and several area churches and served 
young people who read three years or less 
below grade level. 
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Analysis 



Following is a three-part analysis of the evaluations in this compendium focused on program components, 
outcomes and contributing factors. 

Program Components 

Each program evaluation summary in Section II briefly describes the key components of the program. 

This section of the analyses provides two tables that describe key program components and indicates which 
compendium programs use which program components. 



Table I 

Examples of Typical Out-of-School Time Program Components 



Program Component 


Examples 


Academic Assistance 


tutoring; homework assistance; small 
learning groups; reading groups; and 
writing projects 


Cultural Enrichment 


drama; visual arts and crafts projects; 
dance/movement instruction; creative 
writing; field trips to cultural performances; 
and exhibits 


Drug & Alcohol Prevention 


violence and drug prevention initiatives; 




and drug counseling 


Life Skills & Training 


conflict resolution training; health 
education; nutritional education; anger 
management; and peer discussion of 




important topics to youth 


Mentoring 


one-to-one mentoring 


Parent & Community Involvement 


parent support groups and counseling; 
adult education; free after-school child-care; 
parent-child computer classes; voter 
registration drives; community clean-ups; 
parenting classes; and workshops on 
naturalization and related legal issues 


Sports & Recreation 


organized team sports; fitness classes; 
martial arts; and free time for recreational 
play 
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Table II 

Program Components Used By Each Compendium Program 
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Outcomes 



Each program evaluation summary describes the key 
findings from the evaluations, with a focus on youth 
outcomes. This part of the analysis provides an 
overview of the types of outcomes achieved by youth 
participating in these 13 OST programs. The major 
program outcomes for young people are discussed 
below and include: 

• Academic Achievement 

• School Attendance 

• School Engagement and Academic 
Self-Concept 

• Sense of Safety 

• Social and Developmental 

This part of the analysis also includes information 
on the short-term and long-term outcomes that might 
be expected of OST programs. 

Academic Achievement 
Program evaluations examine youth academic 
achievement, using standardized tests, school grades, 
and other measures of academic success. Because of 
increased funding to OST programs along with the 
expectations of some that OST programs should 
improve academic outcomes, the viability of OST 
programs is often determined by evaluation findings 
that demonstrate gains in academic outcomes. 
Programs that do not demonstrate academic outcomes 
for young people may receive cuts in funding or may 
be viewed unfavorably; therefore, one of the most 
important program outcomes for young people is 
academic achievement. 

In this compendium, 11 out of 13 evaluations identi- 
fied academic achievement outcomes for young 
people. In most cases, students who attended OST 
programs were more likely than their non-program 
peers to have higher academic outcomes. For 
example, high school students who attended QOP 
were more likely to graduate from high school and 
to later attend college than comparable students who 
did not attend QOP. 



The evaluation of the TASC program also found 
better academic outcomes for participants than 
non-participants. Evaluators report that on New York 
State Regents Math exams, TASC active participants 
(students who participated in more than 60% of 
after-school programming) were more likely to take 
and pass the Regents Math Sequential I exam by 
ninth grade than were non-participants (students who 
did not participate in the TASC after-school programs, 
but who attended school with TASC students); specif- 
ically, 32 percent of active ninth-grade participants 
passed the exam, compared to 1 percent of non-par- 
ticipant ninth graders. Further, TASC results showed 
that a higher proportion of active participants than 
non-participants took and passed the Regents 
English exam, by the end of eleventh grade (23 
percent, compared to 9 percent). 

CCK and Sacramento START represent city-wide 
initiatives to establish OST programs that would 
improve achievement outcomes or increase access 
to OST programs for young people. Youth, who 
attended CCK , increased their math achievement and 
homework completion grades (a specific category 
listed on student’s reports cards where teachers rate 
students’ performance on out-of-class assignments), 
at a higher rate than youth who did not attend CCK. 
Although the Sacramento START evaluation did not 
examine homework completion grades, the evalua- 
tion found that START students performed signifi- 
cantly better when post-tested on their math SAT 9 
scores. Further, START students who were English 
Language Learners showed greater improvement on 
math achievement than their English-speaking peers. 

The START finding that English language learners 
showed greater improvements than other START 
students was similar to the 21 st CCLC evaluation 
findings. While the 21 st CCLC evaluation found that 
program participants were just as likely to share 
similar learning outcomes as their non-participating 
peers, sub-group analysis for African American stu- 
dents found that students showed increased effort in 
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the classroom, reduced lateness for school, and 
increased math grades as a result of their program 
participation. Impacts on math grades and being on 
time to class were also evident for Latino students. 
None of these impacts were evident for white students. 

Other programs showing improved outcomes for 
minority students were LA s BEST and BELL. LA s 
BEST helped Latino students became more profi- 
cient in English than their peers who did not attend 
the after-school program. Pre- and post- data for 
BELL students revealed that there were significant 
differences in reading and math scaled scores for 
student’s over time. Further, 78% (n=208) of BELL 
students made grade-equivalent gains in reading and 
79% (n=208) made grade-equivalent gains in math 
(n=208). On average BELL students began 5 months 
behind grade-level in reading and 3 months behind 
in math. By the end of the 30-week program, students 
were 1 month above grade-level in reading and 3 
months above grade-level in math. 

A unique faith-based program, YET, was specifically 
set up to improve the literacy skills of young people. 
The results of the YET evaluation showed that after 
students attended the program for 90 days, their 
reading ability improved by at least one-grade level. 

Finally, evaluation results for nationally recognized 
programs, such as BB/BS, 4-H , and BBCA , show that 
students attending these programs improved their 
grade point averages or school grades. 

School Attendance 

An important factor in students’ improved academic 
performance may be increased school attendance. 
Evaluations measure school attendance in a number 
of ways: (1) the number of days students attend 
classes; (2) the number of days students are absent 
from school; or (3) how many days students do not 
cut classes. Although program evaluations have 
shown that participation in programs can be associ- 
ated with increased school attendance, few evalua- 
tions explain or speculate why an extra day in school 
may be important to young people’s development. 
Does an extra one or two days in school for youth 
lead to improved outcomes? Further, how many 



more days in school are critical to improving out- 
comes? These questions need to be answered in 
order to gain a deeper understanding of the effects 
of increased school attendance on youth outcomes. 

In short, evaluations that simply report that youth 
attended one more day of school during the program 
intervention tell us very little about the relevance of 
increased school attendance for young people. Below 
are brief descriptions of programs that reported 
school attendance outcomes. 

In this compendium, eight out of 13 programs, 
reported improved school attendance outcomes for 
youth. Programs like BGCA, 4-H , START, CCK 
showed that youth who attended OST programs 
missed fewer days of school. For example, students 
who attended START programs in the previous year 
improved their school attendance the following year 
from an average of 5.5 days absent to 5.4 days 
absent, a reduction of 0.1 days absent. 

Other programs such as Beacons and BB/BS reported 
that students who attended their programs cut or 
skipped fewer classes. 

TASC examined the attendance rates of low attendees 
(defined as students in the lowest attendance quartile) 
in the year before enrolling in TASC. It was found 
that their attendance rates improved in comparison to 
low attending non-participants. Fifty-one percent of 
the low-attending active participants improved their 
school attendance enough to move out of the lowest 
attendance quartile, while only 32 percent of non- 
participants improved their attendance to the same 
degree. Additionally, the TASC program reported the 
change in school attendance between middle school 
active participants and middle school non-partici- 
pants during one school year program as 1.8 days. 

Finally, the LAs BEST program evaluation speculated 
on the importance of program students’ increased 
school attendance. According to the evaluators, one 
cause of the academic achievement for LA s BEST 
participants was increased school attendance. The 
evaluators charted average annual attendance rates 
for LAs BEST middle school students in 1994/95 
and their peers in 1998/99. LA s BEST students 
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attended school three more days per year than the 
comparison group. 

School Engagement and Academic Self-Concept 
Finn and Voelkl (1993) define school engagement 
as an adolescent’s regular participation in classroom 
and school activities and a young person’s belief 
that school is an important aspect of their life. Three 
out of 13 evaluations found that youth increased 
their school engagement while participating in 
OST programs. 

According to teachers, LA s BEST students were 
more engaged in learning than their peers who did 
not attend the program. Teachers of 4-H students 
reported improvement of classroom behavior of 
elementary students. For teens involved in the 4-H 
program, the mean suspension rate decreased from 
six days (1996) to zero days (1997). 

A contributing factor to student’s school engagement 
is their positive beliefs about their academic abilities. 
Mean pre- and post-test scores indicate that there 
were significant differences found for BELL stu- 
dents’ self-perception of their math ability. 

Sense of Safety 

Although one of the primary foci of OST programs 
is to keep children safe after school, only three out 
of 13 evaluations addressed the issue of youth per- 
ception of safety after school. 

An overwhelming majority of surveyed adolescents 
(86%) reported that they always or often felt safe in 
Beacon programs. 

One of the 21 st CCLC program objectives was to 
make sure that children felt safe after school. 
However, non-program participants were slightly 
more likely to report that they felt safe after school 
(62.1%) than program participants (60.5%). 

By providing a safe and fun-learning environment 
after school, the START program offered an alternative 
avenue of academic enrichment for minority and 
low-income students. 



Social and Developmental 

OST programs fill the critical hours after school for 
adolescents with structured activities and adult super- 
vision. Adolescents that participate in structured 
activities spend less time watching television and 
more time in academic activities and enrichment 
lessons. In contrast, adolescents that spend time in 
unstructured activities, such as television watching 
or hanging out, are placed at risk for poorer social 
and developmental outcomes such as academic 
grades, work habits, and emotional adjustment 
(Posner & Vandell, 1994). Previous research studies 
have shown that participation in OST programs has 
lowered youth delinquency, teenage pregnancy rates, 
and other anti-social behaviors. (Center for Human 
Resources, 1995; Posner & Vandell, 1994; and 
Schinke, Orlandi & Cole, 1992). 

Evaluations of several programs (five out of 13) 
found that program participation was critical to 
improved social and developmental outcomes. The 
BB/BS evaluation reported that participants were 
46 percent less likely to initiate drug use (minority 
Little Brothers and minority Little Sisters were 70 
percent less likely to initiate drug use); 27 percent 
less likely to initiate alcohol use (minority Little 
Sisters were 54 percent less likely to initiate alcohol 
use); and 32 percent less likely to hit someone. 

Surveys of Beacon students showed that youth who 
participated at sites with higher youth development 
quality (i.e., the effective implementation of the 
principles of youth development) compared to 
Beacon sites that had general youth development 
quality (i.e., emphasis on keeping youth safe, well- 
organized activities, low staff-youth ratio) were 
significantly less likely to report that they had hit 
others to hurt them, deliberately damaged other 
people’s property, stolen money or other property, 
and/or been in a fight. 

Compared to control group students, QOP students 
became teen parents less often (24% vs. 38%) and 
they had a third less children. QOP students were 
also less likely to be arrested (19% vs. 23%) and 
they were involved in fewer total crimes. The control 
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group had 50 convictions for every 100 students, 
while the participants had only eight convictions for 
every 100 students, or one-sixth as many convictions. 

Short-Term and Long-Term Outcomes 
The attached table of “Aspects of Compendia 
Programs” provides some information on the types 



of short and long-term outcomes that might be 
expected from OST programs. Most of the evalua- 
tions in this volume only record a short-time period 
of program operation, and, while they report on 
some of the longer-term outcomes, these outcomes 
might have been greater had the evaluations covered 
a longer-time period of program operation. 
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Contributing Factors 



Each program evaluation summary in Section II 
includes a section entitled “Contributing Factors,” 
capturing what the program evaluator speculates may 
be responsible for the youth outcomes achieved by 
the programs. This part of the analysis describes the 
following factors that may have contributed to 
program success in out-of-school time programs: 

• Caring Relationships 

• Community-School Partnerships 

• Developmental Framework 

• Family/Community Involvement 

• Small Learning Communities 

• Staff Quality/Development and Training 

Caring Relationships 

Caring adult relationships are critical to the develop- 
ment of young people. Youth who are in supportive 
relationships with adults are provided with care, 
nurturance, and guidance. Further, caring relation- 
ships with adults provide youth with a mentor or role 
model that can recognize their personal strengths 
and talents and help youth increase these abilities 
and skills (AMHB, 2001). 

Evaluations of seven out of 13 programs reported 
that providing youth with caring adult relationships 
was an important contributing factor to successful 
outcomes for young people. Programs like BELL 
ensure that staff commit to a one-academic-year 
relationship with students. 

Other programs, such as Big Brothers/Big Sisters 
and JUMP , support one-to-one mentoring matches 
between volunteer adults and young people. The 
foundation of both programs is centered on adult 
volunteers providing consistent and caring 
relationships to young people that serve to prevent 
delinquent behavior and promote positive youth 
developmental outcomes. 



' Participation in a [BB/BS] program reduced 
illegal drug activity and alcohol use , began to 
improve academic performance , behavior and 
attitudes , and improved peer and family relation- 
ships. Yet the [BB/BS] approach does not target 
those aspects of life, nor directly address them. 

It simply provides a caring, adult friend.” 
-Public/Private Ventures 

Community-School Partnerships 
There is a growing recognition that collaborative 
partnerships with schools and local city governments 
are essential to help OST programs raise academic 
achievement and to promote positive developmental 
outcomes for young people. Five out of the 13 
programs had collaborative partnerships with local 
governments and schools. 

LAs BEST began as an initiative from the Mayor’s 
office to reduce gang-related activities in the city. 
The partnership between the city government and the 
schools has continued to be crucial to the success of 
the program. “The city has to be a player,” says 
Carla Sanger, the CEO and President of the program, 
“we can’t leave it to school districts alone. There are 
so many resources that a city can make accessible 
and affordable — theaters, parks, and field trips.” 

TASC established many new or enhanced partner- 
ships that created more service venues and addi- 
tional funding. An example of such partnerships 
included a relationship with the New York City 
Board of Education that helped to generate $8.5 
million in expense reimbursements for TASC 
projects and the Madison Square Garden 
Foundation/Cheering for Children Foundation that 
gave students and their families greater exposure 
to New York’s sports and cultural resources. 

JUMP s infrastructure is maintained through an 
array of partnerships with local educational agencies 
(LEA), community-based organizations, schools, and 
project leaders. 
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Developmental Framework 
Many effective programs have moved away from a 
focus on eliminating youth deficits to one of support- 
ing youth assets. In an effort to identify the elements 
of a strength-based approach to healthy development, 
the Search Institute developed the framework of 
developmental assets, which identifies 40 critical 
factors for young people’s growth and development. 
Together, the assets offer a set of benchmarks for 
positive child and adolescent development (Scales 
& Leffert, 1999). Four out of 13 programs reported 
using an asset-based approach to engage 
young people. 

The findings from the BB/BS evaluation speak to 
the effectiveness of an asset-based approach to youth 
policy that is very different from the problem-ori- 
ented approach prevalent in some youth program- 
ming. This developmental approach does not target 
specific problems, but rather interacts flexibly with 
youth in a supportive manner. 

The Beacons ' developmental approach views 
“youth as resources.” In about three-quarters of 
the Beacons , youth are involved in organizing and 
implementing activities and events. Nearly 90% 
of the Beacons have a youth council, 86% involve 
youth as volunteers, and 76% engage youth as paid 
program and administrative staff. Lastly, close to 
60% of Beacons involve youth in community service 
at least once a month. 

Family/Community Involvement 
Collaborative and participatory family or community 
involvement in youth programs can lead to improved 
youth outcomes. Evaluation of six out of the 13 pro- 
grams reported that family or community members 
were involved with program activities. 

Critical to the original vision of the Beacons was 
the creation of a safe place for parents and children 
to partake in an array of services and activities 
designed to strengthen both family and community 
life. Some of these activities include family and 
community holiday celebrations, events honoring 
cultural traditions, parent-support groups, and family 
counseling services. 



Many parents used the CCK program as affordable 
childcare. Parents also served as volunteers and 
worked at CCK sites. CCK reached out to families by 
providing referrals for social services, sending food 
home with children to share with their families, and 
having parents help plan summer program activities. 

Sacramento START consciously worked to involve 
members of the community in its after-school 
program, hiring nearly three-quarters of its staff 
from neighborhoods surrounding the elementary 
schools where the program was held. 

Small Learning Communities 
Some programs used small learning communities to 
improve student outcomes. Small learning communi- 
ties facilitate individualized instruction and greater 
personal contact with caring adults and their peers. 
Four out of 13 programs emphasized engaging stu- 
dents in small learning communities. 

With no more than 20 students per staff member, 

LA s BEST provides for intimate learning opportuni- 
ties in which staff members can give personalized 
instruction and attention to each child. 

Because QOP project sites were small (25 students 
in each site), students were able to bond with each 
other and with adults in the program. 

Staff Quality/Development and Training 
All 13 programs recognized the need for a profes- 
sional and well-planned approached to development 
and training for all staff. The following are program 
examples: 

• The Youth Development Institute (YDI) offers a 
wide range of professional development oppor- 
tunities for both Beacon directors and staff. 

These opportunities include monthly meetings of 
directors to help them incorporate a youth devel- 
opment perspective into organizational behavior 
and access to training where Beacon staff can 
learn the principles and practices of positive 
youth development. 

• Staff members of the BELL program receive 
16 hours of initial training to help them meet 
program objectives. Twenty-four hours of addi- 
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tional training throughout the duration of the 
program is also offered. Training topics include 
use of BELL curricula, instructional practices, 
and mentoring. 

• Sacramento START staff is involved in a number 
of trainings related to program implementation 
process and principles. Core trainings are 
required for all site directors and program 
leaders and consist of workshops in 
classroom/behavior management, enrichment, 
homework/tutoring, literacy, attendance, and 
lesson plans. The START training program is a 
key strategy for aligning after-school program 
activities with the schools’ curriculum. 



• Most TASC site coordinators (91%) were college 
graduates and had at least three years of prior 
experience working in social services, youth 
services, community agencies or educational 
organizations. About half of all site coordinators 
had previously managed one of the above types 
of agencies. Further, about a quarter of after- 
school staff also worked in the host school as 
teachers, classroom aides or instructional assis- 
tants, or other staff. Most said that their dual 
roles benefited both the regular and after-school 
program. 
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Recommendations 



The thorough analysis of the OST programs included 
in this report suggests a number of principles that 
OST programs should follow in order to achieve 
positive outcomes for young people. These principles 
parallel those in other AYPF compendia, as well as 
principles reflected in youth development research as 
a whole. Based on the findings on this report, OST 
programs should have: 

• Caring and well-trained staff and volunteers ; 
Caring adults relationships are critical to the 
development of young people. Programs should 
recruit well-qualified and caring staff and volun- 
teers. Training should provide information on 
youth developmental outcomes, in addition to 
how to implement specific program components. 
Training should also be on-going, and include 
opportunities to evaluate staff performance. OST 
programs should carefully consider their goals 
and objectives and determine what the appropri- 
ate mix of certified classroom teachers, youth 
development professionals and volunteers might 
be, as each brings different skills to the OST 
program. In considering staffing issues, programs 
should keep in mind the need for consistency 
and dependability in the lives of children. 

• Clear goals and objectives. To demonstrate 
effectiveness, OST programs need clear goals, 
defined objectives, and specified outcomes. This 
is important so that OST programs can be evalu- 
ated on the factors they are attempting to influ- 
ence. For example, a program aimed at fostering 
teamwork through soccer should not be held 
accountable for increasing math scores. On the 
other hand, all programs should have ways of 
measuring a wide range of achievement objec- 
tives that may be indirectly influenced by the 
OST activities. 

• Dependable and structured programming. The 
presence of structured programming appeared to 
be the main source of difference between youth 
in treatment and control groups, who, in many 
cases, both participated in some type of activity 
after school hours. The edge provided by formal 
OST programs was often relatively small, related 



to incremental increases in attendance or academic 
performance, and seemed in part attributable to 
the structure of the OST program as opposed to 
unstructured participation in random activities 
occurring after school. 

A consistent schedule of activities also has the 
potential of increasing the involvement of young 
people, particularly middle and high school-age 
youth who can “vote with their feet” and skip 
the OST program when teachers change or their 
activity of interest is not offered on a particular 
day. Programs can then adapt to provide the set 
of activities most attractive to participants. 

• A focus on both academic and developmental 
outcomes. OST programs should not only 
provide academic assistance and keep children 
safe and out of trouble, but also provide engaging 
environments that motivate and inspire learning 
through developmentally appropriate cognitive, 
social, and physical activities. 

• A focus on both short-term and long-term 
outcomes. Most programs developed a “logic 
model” approach that established the connec- 
tions between existing problems or conditions 
that impact youth, program activities, and the 
intended outcome that these activities would 
have on young people. These models indicated 
both the expected short-term results of the OST 
program, such as increasing adult role models 
and structured activities for youth, and the 
longer-term outcomes such as increased school 
attendance, grades and improved developmental 
outcomes. Short-term outcomes properly sus- 
tained and directed should lead to the long-term 
outcomes sought. 

• High quality and continuous evaluation . OST 
programs need to be evaluated to learn more 
about the factors that lead to positive youth 
outcomes, both directly and indirectly, as well 
as, short- and long-term. Rigorous evaluation 
methods should be used wherever possible 
including the use of matched control and 
comparison groups or pre- and post-tests. 
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Evaluation results should be used to continually 
adapt and improve program activities and 
processes. Both process and outcome evaluations 
are useful to the development of effective pro- 
grams. Youth outcomes should be part of the 
evaluation information collected, but caution 
should be used in attributing a variety of youth 
outcomes to OST programs. Many other factors 
are related to both positive and negative outcomes 
for young people, most prominently the quality 
of each child’s formal schooling experience and 
family situation. 

• Inclusion of families in programs . OST programs 
can benefit from the inclusion of families in pro- 
grams. Building communication and interactions 
with parents can increase student involvement. 
Parents can also be involved in planning various 
OST activities such as cultural and recreational 



activities. In addition, some OST activities 
can be directed at parents including English 
Language Development or home ownership 
classes held after school at the same location 
as the OST program for their children. 

• A focus on youth development including 
increasing youth leadership . The positive youth 
outcomes of OST programs were sometimes 
under-realized due to low participation by young 
people who did not attend on a daily basis. 
Middle and high school-aged youth are those 
most likely to have competing after-school 
options and the ability to decide how to spend 
their after-school time. For these young people, 
in particular, involving them in planning OST 
activities may be essential to holding their attention 
and increasing their participation and outcomes. 
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4-H: Kansas City, MO 



A Summary of: 

“4-H as an Urban Program” 

(1998), Resource Development Institute 



Program Activity 

/ Academic assistance 
/ Cultural enrichment 
/ Drug & alcohol prevention 
/ Life skills & training 
/ Mentoring 

/ Parent & community involvement 
/ Sports & recreation 



Overview 

The largest voluntary co-educational youth program 
in the world, 4-H is implemented in both rural and 
urban settings. The four H’s stand for head, heart, 
hands, and health and reflect the program’s pledge: 
“My head to clearer thinking; my heart to greater 
loyalty; my hands to larger services; and my health 
to better living ” Since 1914, 4-H programs have 
been administered by Cooperative Extension offices 
formed by partnerships among federal, state, and 
county governments and other local public and 
private organizations. The Extension office must 
include a university, which is almost always the land- 
grant university. The program was initially developed 
to improve the living conditions of small farmers 
and their families through education and was focused 
on school-aged children. In the late 1970s, the 
concept was expanded and introduced in urban 
settings. The first after-school 4-H program in public 
housing started in Los Angeles. In 1995, with the 



Population 

In April 1996, 4-H opened three public 
housing developments in Kansas City, MO to 
serve 40 youth. Two other sites were opened 
the next year for a total enrollment of 145 
youth and an average daily attendance of 
120. There were six more girls than boys in 
the group. Ages vary from 5 to 11 years old, 
with the largest group between 5 and 7 
years of age. Ninety-eight percent of the 
145 youth were African Americans. Eighteen 
teenagers, also public housing residents, 
worked as mentors. 

support of the Department of Housing and Urban 
Development, this urban model was replicated in 
other cities. This report focuses on programs in 
Kansas City. 



Key Findings 

The program maintained a continuous evaluation 
system that focused on both process and outcomes for 
the child, the family and the community. A compari- 
son of students’ performance prior to and after partici- 
pation in the program indicated that 4-H participation: 

• increased school attendance (school attendance 
increased from an average of less than two days 
to more than four days per week for participants 
in elementary school) 



• improved classroom behavior (teachers reported 
improved classroom behavior of elementary students; 
for teens involved in the program, the mean suspen- 
sion rate decreased from six to zero) 

• improved grade point average (teens’ grade point 
averages increased as much as three grade points, 
and all students met appropriate grade level standards; 
grade improvement in elementary school children 
was directly related to intensity of participation in 
the program) 
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• improved behavior at home (parents and community 
leaders reported that participants demonstrated 
improved relations with siblings and parents) 

• reduced illegal activities in the community (at least 



six of the teens had been involved with the police 
before; after entering the program all 18 teens and 
the 145 elementary students had no reported partici- 
pation in drug use or other illegal behaviors) 



Program Components 

All 4-H Programs emphasize a “hands on” approach 
and the connections between academic and work- 
related learning, with emphasis on life skills devel- 
opment and a strong community-centered focus. 

The programs also provide youth with opportunities 
to interact with adults and other successful teens 
beyond the school environment. 

The Kansas City 4-H program is an after-school 
program that includes: 

• educational enrichment (organized in small groups, 
participants work daily on a curriculum especially 
developed for 4-H, which focuses on reading, 
vocabulary, ethnic enrichment, science, math and 
entrepreneurial topics) 

• employment of local residents as site director and 
mentors (each site director supervises up to seven 
teenage mentors; all staff must reside in the public 
housing where the program is located) 

• on-going staff development (staff receive a 
minimum of 250 hours per year of training in 
curriculum development, personal development, 
conflict resolution, interpersonal relations, job skills 
and career development) 



• customized curricula (character building and drug 
prevention strategies are integrated with academic 
curriculum and individualized to meet the needs of 
participant youth and families) 

• nutritional education (participants learn about eating 
healthy food and a nutritional snack or meal is 
served each day; for some 4-H participants, this 
may be their only evening meal) 

• mentoring (participants are asked to bring all 
homework to the site and are helped by teen 
mentors when needed; teen mentors are local resi- 
dents selected for their success in school and their 
ethical and moral behavior; the average ratio is six 
participants per mentor) 

• community activities (the 4-H concept is that the 
more a family is involved in the community, the 
healthier the behavior of its members; community 
activities are used as a venue for recognizing 
achievements in academics, sports, and ethical and 
moral behaviors) 

• school connection (includes collaborative and 
coordinated program planning, ongoing school 
visitation to monitor attendance and academic 
growth, advocacy for children and youth) 



Contributing Factors 

Community Involvement 

Site residents are actively involved in the programs 
and may serve in the local Resident Management 
Councils, a part of public housing management. 
They also serve on the Vision Team, the program’s 
advisory board for operations and expansion. The 
Vision Team is composed of representatives of the 
founding Coalition, private industries, local, state 
and federal government agencies, and site residents, 
including teen mentors and project participants. 

On-going Evaluation 

A system of continuous evaluation and feedback is 
used to monitor the program and its outcomes while 



also providing information that is used by the site’s 
Management Council and Vision Team to improve 
and modify the programs. 

Holistic Approach 

The 4-H concept focuses on the individual, the 
family and the community. The programs equally 
emphasize success in academics, sports, work, 
respect for others, and dedication to the community. 

Contact with Caring Adults 

Many of the participating children are latchkey chil- 
dren, who see the 4-H staff as a steady support and 
their main role models. 
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Study Methodology 

Kansas City 4-H maintains a system of on-going 
evaluation that combines qualitative and quanti- 
tative methodologies. Evaluation includes 
process (achievement of goals and objectives) 
and outcomes. Data is used to improve the 
program. School attendance and grades are 
collected from the schools and 4-H attendance 
is monitored by the site directors. School-based 
behavioral data is collected from teachers using 
the Walker-McConnell Scale. Behavior in the 
family and community was measured by surveys 
with parents and community leaders. 



Evaluation Funding 

Grant from the U.S. Department of Housing and 
Urban Development. 

Geographic Areas 

Kansas City, MO 

Contact Information 

Leon A. Moon, Project Director 
Resource Development Institute 
University Outreach and Extension 
University of Missouri 
PO Box 270304 
Kansas City, Missouri 64127 
Phone: (816) 221-3383 
Fax: (816) 842-6920 
MoonL@missouri.edu 
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21 st Century Community Learning Centers 



A Summary of: 

“When Schools Stay Open Late: The 
National Evaluation of the 21 st Century 
Community Learning Centers Program” 

(2003) by Mathematica Policy Research, Inc. & 
Decision Information Resources, Inc. 



Program Activity 

✓ Academic assistance 
/ Cultural enrichment 

✓ Drug & alcohol prevention 

✓ Life skills & training 
Mentoring 

✓ Parent & community involvement 
/ Sports & recreation 



Overview 

In 1994, the 21 s * Century Community Learning 
Centers (21 st CCLC) program was authorized by 
Congress to open up schools for broader use by 
communities. The 21 st CCLC offered an opportunity 
for students and their families to continue to learn 
new skills and discover new abilities after school. 

In 1998, the program was refocused on supporting 
schools to provide school-based academic and recre- 
ational activities after school and during other times 
when schools were not in session, i.e., weekends and 
holidays. Programs were funded on a competitive 
basis through federal grants made directly to local 
educational agencies (LEAs). The No Child Left 
Behind Act of 2001 changed the 21 s1 Century 
program further by allotting funds directly to states 
to carry out their own grant competitions and awards 
and allowing education agencies as well as community 
and nonprofit organizations to run CLCs. 

The evaluation acknowledged and measured three 
important goals of the 21 s ' CCLC programs for the 
first three cohorts 1998-1999, 1999-2000, 2000-2001: 
academic achievement outcomes, developmental 
outcomes and creating safe places for children. 

One of the most important components of the 21 s ' 
CCLC’s was the program’s focus on academics. In 
21 s ' CCLC after-school programs, tutorial services 
and academic enrichment activities were offered to 
help students meet local and state academic standards 
in reading and math. 



Population 

Elementary School Study Population 

There were 587 program students and 381 
comparison students that participated in this 
study. Of the program students that participated 
in this study, 47% were males and 54% were 
females. Ten percent of the program students 
were white, 67% African American, 19% Latino, 
and 2% Other. Of the non-program students, 
51% were males and 50% were females. Eight 
percent of the students were white, 73% African 
American, 14% Latino, and 2% Other. 

Middle School Study Population 

There were 1,752 program students and 2,437 
comparison students that participated in this 
study. Of the program students that participated 
in this study, 47% were males and 53% were 
females. Thirty-eight percent of the program 
students were white, 28% African American, 
12% Latino, 16% other and 6% were Mixed 
Race. Thirty percent of program participants 
earned grades of mostly A’s; 36% of mostly 
B’s, 23% of mostly C’s, 9% of mostly D’s or 
below, and 2% were not graded. Of the non- 
program students, 47% were males and 54% 
were females. Forty-one percent of the students 
were white, 25% African American, 12% 

Latino, 16% Other, and 7% Mixed race. Thirty- 
four percent of non-program students received 
grades of mostly A’s, 36% mostly B’s, 21% 
mostly C’s, 8% mostly D’s or below, and less 
than 1% were not graded. 
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21 st CCLC programs that participated in the evalua- 
tion were housed in elementary and middle schools 
(high school students were served by some of the 
programs but were not included in the evaluation). 
Most public school grantees that were part of the 
study had operated some type of after-school 
program before receiving a 21 st Century grant and 



were using their grant funds to expand or modify 
their services and activities. Funding for the 21 st 
CCLC grew from an appropriation of $40 million 
in fiscal year 1998 to $1 billion in fiscal year 2002. 
Financial support for 21 st CCLC programs supports 
after-school programs in about 7,500 rural and inner- 
city public schools in more than 1,400 communities. 



Key Findings 

Elementary School Students 

Learning Outcomes 

21 st CCLC program students had similar learning 
outcomes as non-program students on reading 
(82.6 vs. 81.7) and math grades (81.0 vs. 79.6) 
and reading test scores (34.3 vs. 34.1). However, 
program student’s social studies grades did differ 
significantly from non-program peers (83.0 vs. 80.0). 

Parental Involvement 

Parents of elementary program students were more 
likely to help their child with homework or ask about 
things they were doing in class. 

Sense of Safety 

One of 21 st CCLC program objectives was to make 
sure that children felt a sense of safety after school. 
Although the difference between the two groups was 
not significant, program students reported that they 
were less likely to feel safe after school than non- 
program students (74.3 vs. 75.5). 

Anti-social Behavior 

The number of suspensions, school absences, and 
teacher reports of discipline problems were similar 
for both 21 st CCLC program students and non- 
program students. 

Developmental Outcomes 

21 st CCLC program participants were no more likely 
than their counterparts to report getting along with 
others their age, feeling included, being good at 
working with others in a team, or setting a goal 
and working to achieve it. 



Middle School Students 

Learning Outcomes 

Evaluators highlighted that improving learning out- 
comes is what distinguished 21 st CCLC after-school 
programs and an overwhelming number of parents 
(i.e., 75%) believed that program participation would 
help improve their child’s school grades. However, 
program participants were just as likely to share 
similar learning outcomes as their non-participating 
peers on their English, science, and social studies or 
history grades. Program participants were as likely 
to complete their homework as their non-participat- 
ing peers, although they were more likely to do so 
to their teachers’ satisfaction. Participants did have 
slightly higher math grades and school attendance. 

A subgroup analysis found that more frequent atten- 
dee (4-5 days per week) did not have larger learning 
gains than less frequent attendee (2 days per week). 
A subgroup analysis comparing programs that put 
more or less emphasis on academic achievement also 
showed no differences in achievement gains. 

Improved student outcomes for African Americans 
and Latinos 

Sub-group analysis for African American students 
found that students showed increased effort in the 
classroom, reduced lateness for school and increased 
math grades as a result of their program participation. 
Impacts on math grades and being on time to class 
were evident for Latino students. None of these 
impacts were evident for white students. 

Parental Involvement 

Research on middle school students showed that 
parents increased their involvement at student’s 
schools. Parents were more likely to volunteer at 
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their child’s school and attend open houses or parent- 
teacher organization meetings. 

Sense of Safety 

One of the 21 st CCLC program objectives was to 
make sure that children felt a sense of safety after 
school. However, non-program participants were 
slightly more likely to report that they felt safe after 
school (62.1%) than program participants (60.5%), 
although this result did not differ significantly 
between participants and nonparticipants. 

Anti-social Behavior 

Although rates were not high, program participants 
were more likely to participate in anti-social behav- 
ior than their non-program counterparts, i.e., selling 
drugs (3.3% vs. 1.8% at the .05 significance level) 



or smoking marijuana, and female program partici- 
pants were more likely to have had their personal 
property damaged or to have been picked on after 
school. Other measures of behavior — such as receiving 
detention for misbehavior, being sent to the office, 
and teacher reports of discipline problems— were 
the same in both groups. 

Developmental Outcomes 

21 st CCLC program students reported similar scores 
of being able to get along with others their age, feel- 
ings of being included, being good at working with 
others in a team, or setting a goal and working to 
achieve it. Middle school program participants were 
less likely than nonprogram participants to rate 
themselves as good or excellent at working out 
conflicts with others. 



Program Components 

Academic Assistance 

Generally, students received 45 minutes to an hour of 
academic assistance as their first activity in program 
sites. This was also the most prevalent activity at sites. 
For many sites, this took the form of homework assis- 
tance. The homework sessions typically consisted of 
20 students monitored by two staff members (usually 
certified teachers or a certified teacher and a parapro- 
fessional). Despite the presence of teachers at host 
school homework sessions, site visitors rarely 
observed staff members checking homework for com- 
pleteness and accuracy. Further, homework sessions 
were often noisy. A few centers did develop strategies 
to strengthen homework sessions, i.e., using journals 
to find out what students were supposed to be 
working on and sending notes to the regular teacher 
that the student had completed the assignment. 



Recreation 

Recreational activities include swimming, weight 
training, bowling and basketball, among others. This 
is the second most prevalent activity offered by after- 
school sites. 

Cultural and Interpersonal Skills 

Most centers (77%) offer cultural activities, such as 
foreign languages, arts and crafts, manners training, 
drama, music and interpersonal skills development 
(leadership, conflict-resolution, and positive peer 
modeling). 

Other 

In addition, 21 st CCLC programs provide activities, 
such as drug and violence prevention programs, and 
technology education programs. 



Contributing Factors 

Limited Efforts to Form Partnerships and Plan 
for Sustainability 

While programs often worked with community 
organizations (89% of grantees in the study reported 
working with community organizations), the domi- 
nant type of collaboration was for centers to contract 
with community agencies to provide specific after 
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Low Levels of Student Participation 
The average student attended the program less than 
two days a week, despite the fact that programs were 
open to participants four to five days a week. 
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school instruction rather than as partners with shared 
governance or combined operations. Since many 
after-school sites were relying on funding from the 
21 st CCLC grant, it is very important for sites to 
identify additional sources of funding to maintain 
sustainability once 21 s ' Century funds end. At the 
time of site visits, evaluators found that one-third of 
grantees had made no plans and taken no actions to 
sustain their programs; half had developed some 
plans but had not yet taken any action. 

Staff Quality 

A third of the program coordinators and three out of 
five program staff members are school-day teachers. 
To accommodate the varying schedules and require- 



ments of teachers, staff members often worked only a 
few days a week and for short periods. 

Professional Development 
Many program directors and coordinators believe 
that training for staff was unnecessary because many 
staff members had teaching backgrounds and were 
sufficiently trained for performing their after-school 
roles. During the 2000-2001 school year, 75% of 
center coordinators and 25% of staff reported receiv- 
ing training. There are two types of training avail- 
able: (1) orientation (i.e., objectives, policy, and 
procedures of the program) and (2) skill training 
(i.e., taught staff how to perform tasks critical to 
their center roles). 



Study Methodology 

The evaluation’s design includes both a middle 
and elementary school study. The elementary 
school study used random assignment of 
students to treatment and control groups and 
involved 14 school districts and 34 centers. 
Results presented here are from seven school 
districts selected in the first year of the study. 
These school district sites were selected 
because they were over-enrolled and had 
enough applicants to create control groups. 

While the measures had “internal validity,” 
researchers cautioned that the sites are not rep- 
resentative of all 21 st CCLC elementary schools 
in that the sites are more urban, serve a larger 
percentage of minority children and serve more 
disadvantaged children than other elementary 
school programs. 

The middle school study is based on a nationally 
representative sample of after-school programs 
and participants and a matched group of stu- 
dents that is similar to the program participant 
group. Thirty-four districts and 62 centers in the 
districts are included in the middle school study. 
Regression analysis was used to adjust for 
baseline differences between program partici- 
pants and similar students used for comparison 
with variables including demographics, socioeco- 



nomic status, test scores, attendance, 
disciplinary problems and self-reported grades. 

Survey data was collected on family background, 
after-school activities, school experiences, in-school 
and out-of-school behavior, and experiences in 
and knowledge of after-school programs. 

Evaluation Funding 

The evaluation’s primary funding was from the 
U.S. Department of Education, additional funding 
was also received from the C.S. Mott Foundation. 

Contact Information 

Program Contact 
Mark Dynarski 

Mathematica Policy Research 
PO Box 2393 
Princeton, NJ 08543-2393 
Phone: (609) 275-2397 
mdynarski@mathematica-mpr.com 

Evaluation Contact 
Amanda Clyburn 

U.S. Department of Education, OESE 
21st Century Community Learning Centers 
400 Maryland Ave., S.W. 

Washington, DC 20202-6100 
Phone: (202) 260-0919 
amanda.clyburn@ed.gov 
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The After-School Corporation 



A Summary of: 

“Patterns of Student-Level Change Linked 
to TASC Participation, Based on TASC 
Projects in Year 2 ” (2001), by Richard N. White, 
Elizabeth Reisner, Megan Welsh, & Christina Russell 



Program Activity 

/ Academic assistance 
✓ Cultural enrichment 
Drug & alcohol prevention 
/ Life skills & training 
/ Mentoring 

/ Parent & community involvement 
/ Sports & recreation 



Overview 

In an effort to increase the quality and availability 
of after-school programs in New York, the Open 
Society Institute (OSI) established The After-School 
Corporation (TASC). Since the 1998 inception of 
TASC, OSI has committed up to $25 million a year 
to after-school programming for a period of five 
years. This commitment was conditional on TASC 
obtaining matching funds from public and private 
sources on a 3:1 basis. Currently, OSI has extended 
funding to TASC for two additional years and possibly 
beyond. With funding from OSI and collaborative 
support from the New York City Board of Education 
and other sources, TASC’s mission is to start or 
expand school-based after-school programs that are 
operated by non-profit or community-based organi- 
zations (CBOs). To date, TASC has funded more 
than 130 CBOs and other non-profit organizations in 
143 New York City public schools and 75 schools in 
New York State for approximately 40,000 school 
children served by TASC. 



Population 

Students who attended TASC programs were 
typically considered at-risk for educational 
failure (i.e., 68% and 66% of active students 
scored below grade in level in reading and 
math, respectively prior to entering TASC 
programs). Demographic characteristics of 
TASC students included information on active 
participants (i.e., TASC students that attended 
at least 60% of the after-school program). In 
general, program participants were Latino 
(46%), African American (40%), Asian/Pacific 
Islander (9%), white (6%), English Language 
Learners (16%) and Recent Immigrants (8%). 
Forty-eight percent of active TASC students 
were male and 52% females. A significant 
proportion of active students (87%) were 
eligible for free lunch and 6% were eligible 
for reduced-price lunch. TASC programs 
were implemented in K-12th grades. 



Key Findings 

The TASC evaluation compared active participants 
(students who participated in more than 60% of 
after-school programming), non-active participants 
(students who participated in less than 60% of after- 
school programming) and non-participants (students 
who did not participate in the TASC after-school 
programs, but who attended school with TASC stu- 



dents) on the outcome factors of school attendance 
and scores on New York’s Regent exams. 

School Attendance 

K-8 grade. For students in K-8 grade, active partici- 
pants attended school more frequently (92.2% of 
school days) than non-participants (90.9%) in the 
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year prior to their TASC participation. After partici- 
pating one year in the TASC program, active partici- 
pants’ attendance increased to 93.2%, compared to 
no change in the initial rate for non-participants 
(90.9%). The change in school attendance between 
active participants and non-participants, during one 
school year program was 1.8 days. 

10-12th grade. Active students in grades 10-12 
attended school at a rate of 96% in the year before 
their TASC participation compared to non-partici- 
pants at a rate of 93%. In a year after TASC partici- 
pation, both active participants’ and non-participants’ 
attendance rates declined to 93.5% and 89.5%, 
respectively. There was a faster rate of decline for 
non-participants than active participants. The atten- 
dance rates for 9th graders were not included, 
because of inconsistent and missing data. 

Low attendees. When examining the attendance 
rates of low attendees (defined as students in the 
lowest attendance quartile) in the year before 
enrolling in TASC, it was found that their attendance 
rates improved in comparison to low-attending non- 
participants. Fifty-one percent of the low-attending 
active participants improved their school attendance 



enough to move out of the lowest attendance quartile, 
while only 32% of non-participants improved their 
attendance to the same degree. 

Regents math exams. Evaluators reported that 
active TASC participants were more likely to take 
and pass the Regents Math Sequential I exam by 
ninth grade than were non-participants (32% of 
active ninth-grade participants passed the exam, 
compared to 1% of ninth-grade non-participants). 
The Math Sequential I exam is usually administered 
to students who have taken Algebra I. As Algebra I 
is considered to be a gatekeeper course determining 
a student’s eventual readiness for college, it is an 
especially important measure of achievement. In 
comparing active and non-participants, it was deter- 
mined that active participants were more likely to 
advance to Math Sequential 2 and 3 exams (52% of 
active participants passed both the Math Sequential 
2 and 3 exams, compared to 15% of non-participants 
in the same grades). 

Regents English exams. A higher proportion of 
active participants than non-participants took and 
passed this exam by the end of eleventh grade (23%, 
compared to 9%). 



Program Components 

Academic and cognitive development activities 
TASC sites provided students with homework help, 
organized writing and reading activities, and math 
and word games. Older TASC students engage in 
preparatory courses for the Regents and/or SAT 
exams and field trips to high school and college 
campuses. 

Artistic development 

Many opportunities are given to TASC students 
to experience the arts and develop artistic skills 
i.e., visual arts and crafts instruction/projects, 
dance/movement instruction, creative writing 
and field trips to performances and exhibits. 

Development of health , well-being and life skills 
TASC sites report offering the following activities in 
various degrees to promote the health and well-being 



of students: conflict resolution training, health edu- 
cation, instruction in life skills, and peer discussion 
of topics important to youth. 

Sports and recreation 

Most sites offer some form of recreation and sports 
activities i.e., organized team sports, fitness classes, 
martial arts and free time for recreational play. 

Civic engagement and community service 
TASC students learn to relate the world outside their 
school and families through participation in service 
projects, discussion of current events, and opportuni- 
ties to engage in mock govemment/election activities, 
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Contributing Factors 

Staff quality 

Most site coordinators (91%) are college graduates 
and have at least three years of experience working 
in social services, youth services, community agen- 
cies or educational organizations. About half of all 
site coordinators have managed one of these types 
of agencies. Further, about a quarter of after-school 
staff also worked in the host school as teachers, 
classroom aides or instructional assistants, or other 
staff. Most said that their dual roles benefited both 
the regular and after-school program. 

Collaborative partnerships 

TASC established many new or enhanced partnerships 
that created more service venues and additional 
funding. An example of such a partnership included 
one with the New York City Board of Education, 
which generated $8.5 million in expense reimburse- 
ments for TASC projects and the Madison Square 
Garden Foundation/Cheering for Children Foundation 
which gave students and their families greater expo- 
sure to New York’s sports and cultural resources. 



Technical assistance 

TASC provides a few small to mid-size CBO’s with 
intensive technical assistance to increase their ability 
to operate high-quality after-school programs that 
would have a high likelihood of being sustained. 
There is a concern that small to mid-size CBOs 
might not be able to attract outside financial support 
to meet the required mandated funding match or be 
able to bring their programs to sufficient scale to 
support stable operations (proportional to their type 
and level of need). 

Teach after Three 

The Teach after Three program is designed to 
identify and recruit new teaching candidates from 
TASC after-school program sites. TASC, in joint 
partnership with the City University of New York, 
helped recruit several project staff into the New York 
City Teaching Fellows program. TASC also sought 
to develop an alternative training model that would 
place student teachers in after-school sites in high- 
need schools and districts. 



Study Methodology 

Evaluators compared active participants, non- 
active participants and non-participants on the 
outcome factors of school attendance and New 
York Regent exam scores. Pre/post outcome 
data was collected for students one year prior to 
participation in the TASC program (1998-1999) 
and one year after participation in the TASC 
program (1999-2000). 

Evaluation Funding 

C.S. Mott Foundation, Carnegie Corporation of 
New York and W.T. Grant Foundation. 

Geographic Areas 

New York City, NY 



Contact Information 

Research Contact 

Elizabeth Reisner 

Policy Studies Associates 

1718 Connecticut Avenue, Suite 400 

Washington, DC 20009 

Phone: (202) 939-9780 

ereisner@policystudies.com 

Implementing Contact 
Lucy Friedman, President 
TASC 

925 Ninth Avenue 
New York, NY 10023 
Phone: (212) 547-6950 
Fax: (212) 547-6783 



Note: The latest TASC evaluation, not available before press time, includes one year of additional data: What Have We Learned From TASCs First 
Three Years? Evaluation of the TASC After-School Program. Policy Studies Associates, Inc. 
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Beacons: New York City 



A Summary of: 

“Evaluation of the New York City 
Beacons” (2002) by Constancia Warren, Michelle 
Feist & Nancy Nevarez 



Program Activity 

/ Academic assistance 
/ Cultural enrichment 
/ Drug & alcohol prevention 
/ Life skills & training 
Mentoring 

/ Parent & commuity involvement 
/ Sports & recreation 



Overview 

Beacons are community centers located in public 
school buildings that offer a range of activities and 
services for participants of all ages. The centers are 
open before and after school, in the evenings and on 
the weekends. Individual Beacons are managed by 
community-based organizations that work collabora- 
tively with their host schools, community advisory 
councils, and a wide range of neighborhood organi- 
zations and institutions. 

The Beacon initiative started in 1991 with funds 
from the New York City Department of Youth and 
Community Development. The Youth Development 
Institute of the Fund for the City of New York pro- 
vides ongoing support and technical assistance to the 
Beacons, including funding and staff training oppor- 
tunities, linkages to resources, and grants to develop 
specific projects. There are 80 Beacons, with at least 
one operating in each of the 32 local school districts 
in New York City and several in the city’s poorest 



Population 

In FY 1998, more than 77,000 youth (21 
years of age and below) and 36,000 adults 
participated in the 40 Beacons. In fall 1999, 
a total of 231 youth completed the youth 
survey. Youth surveyed were between the 
ages of 12-19. Fifty-three percent of the youth 
surveyed were females and 47% were males. 
Most students were African American (46%) 
and Latino (44%) and fewer than 10% were 
white, American Indian or of mixed heritage. 

neighborhoods. The individual Beacon programs 
offer children, youth, and adults a wide range of recre- 
ational programs, social services, educational enrich- 
ment and vocational activities in several areas: youth 
development programming, academic support and 
enhancement, parental involvement, family support, 
and neighborhood safety and community building. 



Key Findings 

Evaluators examined both general and youth-devel- 
opment quality. The former included safety, well- 
organized activities, consistent enforcement of the 
rules, and low staff-youth ratio. The latter included 
the five elements of good youth development pro- 
gramming that is central to the Youth Development 
Institute’s framework: (1) develop caring and trusting 
relationships, (2) participate in stimulating and 



engaging activities, (3) benefit from a community 
of adult support, (4) be challenged to grow by high 
expectations, and (5) connect with and contribute to 
their communities. Surveys of Beacon students showed 
that youth who participated at sites with “higher youth 
development quality” compared to Beacon sites that 
had “lower youth development quality” were signifi- 
cantly less likely to report that they had: 
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• cut classes, • stolen money or other property, and/or 

• hit others to hurt them, • been in a fight. 

• deliberately damaged other people’s property, 



Program Components 

Beacons differ in the services they offer. However, 
most offer some of the following: 

• recreation activities, 

• adult education (GED prepatory, basic literacy and 
English as a Second Language classes), 

• free after-school child-care, 

• leadership development, 

• parent support groups and counseling, 

• substance abuse and pregnancy prevention activities, 



social services (referral to health and mental 
health services, drug counseling), 

educational enrichment (homework help, reading 
groups, writing projects), 

intergenerational activities (holiday celebrations, 
parent-child computer classes), 

community services (voter registration drives, 
community clean-ups, cultural events), and 

immigrant support services (workshops on natu- 
ralization and related legal issues). 



Contributing Factors 

A Safe and Engaging Place 

Youth feel safe and engaged at the Beacons. Among 
the activities that attract youth are basketball, karate, 
computer instruction, conflict-resolution training, 
newspaper production and leadership development. 

Experienced Staff 

More than three-quarters of Beacons staff have at 
least three years’ experience in the field of youth 
development, and almost half worked for Beacons 
for more than three years. This low staff turnover 
enables youth who use Beacons to build stable and 
caring relationships with staff. 

Staff Development 

The Youth Development Institute (YDI) offers a wide 
range of professional development opportunities for 
both Beacon directors and staff. These opportunities 
include monthly meetings of directors to help them 
incorporate a youth development perspective into 
organizational behavior and access to training in the 
principles and practices of positive youth develop- 
ment. Attendance of professional development train- 
ings and meetings is optional for both staff and direc- 
tors. The sites with staff attending the most training 
opportunities, however, had the highest-rated youth 
development quality and the most positive results. 



High Expectations 

In the majority of activities observed by the evaluators, 
staff challenged youth to ask questions and examine 
their thinking. Seventy-four percent of Beacon youth 
reported that staff have high expectations of their 
conduct and performance. 

Youth as Resources 

In about three-quarters of the Beacons, youth are 
involved in organizing and implementing activities 
and events. Nearly 90% of the Beacons have a youth 
council, 86% involve youth as volunteers and 76% 
engage youth as paid administrative and program 
staff. Sixty percent of all Beacon programs place 
youth in community service at least once a month. 

Family/Community Involvement 
Critical to the original vision of the Beacons is 
to create a safe place for parents and children to 
partake in an array of services and activities to 
strengthen both family and community life. Some 
of these activities include family and community 
holiday celebrations, events honoring cultural tradi- 
tions, parent-support groups, and family counseling 
services. 
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Study Methodology 

Evaluators used and analyzed focus groups, 
interviews, and surveys 1 of Beacon youth, staff, 
administrators, and parents. Site visits were also 
used. A scale was developed to rate the quality 
of youth development at each Beacon site. 

Evaluation Funding 

Annie E. Casey, Ford Foundation and the 
Open Society Institute. 

Geographic Areas 

New York, New York 

Contact Information 

Research Contact 
Michelle Feist 
Program Officer 

Academy for Educational Development 
1 00 Fifth Avenue 
New York, NY 10011 



Implementing Contact 
Peter Kleinbard 
Fund for the City of New York 
Vice President and 

Director of the Youth Development Institute 

Fund for the City of New York 

121 Avenue of the Americas 

New York, N.Y. 10013-1590 

Phone: (212) 925-6675 

Fax: (212) 925-5675 

pkleinbard@fcny.org 

For copies of the report, please contact: 
Elayne Archer 

Academy for Educational Development 
Senior Program Officer 
Phone: (212) 243-1110 
Fax: (212) 627-0407 
earcher@aed.org 



'A survey was developed specifically for Beacon programs to examine youth educational attitudes such as, perceived importance of doing well in school and anti-social 
behavior (i.e., delinquency) and civic activities and attitudes (i.e., participation in volunteer work). 



O 

ERIC 



American Youth Policy Forum 

39 



BESTCOPY AVAILABLE 



Finding Fortune in Thirteen Out-of-School Time Programs 

BELL After-School Instructional Curriculum 



A Summary of: 

“BASICs Afterschool Program 2001-2002 
Academic Year” (2001), by Tiffany M. Cooper 



Program Activity 

/ Academic assistance 
/ Cultural enrichment 
Drug & alcohol prevention 
/ Life skills & training 
/ Mentoring 

/ Parent & community involvement 
Sports & recreation 



Overview 

In 1992, two members of the Harvard Black Law 
Student Association founded the Building Educated 
Leaders for Life (BELL), a community-based, non- 
profit organization. The founders were concerned 
with the lack of quality youth programming for 
black and Latino children living in low-income 
communities. Building on this concern, they devel- 
oped two out-of- school time programs, known as 
the BELL After-School Instructional Curriculum 
(BASICs) and the BELL Accelerated Learning 
Summer Program (BALSP). Both programs combine 
BELL’s efforts to provide year round educational and 
social opportunities to elementary school children, 
particularly those from low-income, under-resourced 
neighborhoods. This evaluation summary will only 
describe the BASICs after-school program. The 
BASICs program meets for three hours after every 
school day, for 30 weeks, offering students concen- 



Population 

The BASICs program served 850 children, 
during the 2001-2002 school year. Programs 
operated in 10 elementary school sites in 
Boston, two sites in New York City and, two 
sites in Washington, DC. There were 208 
students that participated in the evaluation. 

An overwhelming number of these children 
entered the program with reading (83%) and 
math scores (82%) below grade level. 

trated small group tutoring and mentoring. The 
BELL organization believes in the ideal of a fair and 
equitable society, and believes that its efforts in pro- 
viding additional educational opportunities for chil- 
dren will help transform their credence into a reality. 
The after-school program operates in three locations: 
Boston, New York City, and Washington, DC. 



Key Findings 

A cadem ic Achievem ent 

Pre- and post-student data were used to determine 
students reading and math achievement, academic 
self-concept, and school engagement. Portions of the 
Stanford Diagnostic Reading Test-IV (SDRT-IV) and 
the Stanford Diagnostic Math Test-IV (SDMT-IV) 
were used to determine students’ strengths and 
weakness in both areas. Results from SDRT and 
SDMT pre- and post-tests are presented in the form 



of scaled scores, norm curve equivalents, and grade 
equivalent scores: 

• There were significant differences in reading 
(from 581.61 to 596.39; p<001) and math (from 
561.28 to 583.79; p<.001) scaled scores for stu- 
dents. The range of possible scores in reading and 
math was 200 to 800. 

• NCE math means (from 35.32 to 38.16; p<05) 



American Youth Policy Forum 



Finding Fortune in Thirteen Out-of-School Time Programs 



showed significant differences on students’ pre- 
and post-scores. Although there were no signifi- 
cant differences found for NCE reading means 
(from 38.49 to 38.82; p<.001), students’ scores 
did show positive reading gains. The range of 
possible NCE scores was 1 to 99; NCE scores 
were supposed to approximate percentiles. 

• Seventy-eight percent (n=207) of scholars made 
grade-equivalent gains in reading and 79% made 
grade equivalent gains in math (n=208). The 
average gains for reading and math were 5 months 
and 7 months respectively. On average, students 
began BASICS 5 months behind grade-level in 
reading and 3 months behind in math. By the end 
of the 30-week program, scholars were 1 month 
above grade-level in reading and 3 months above 
grade-level in math. 



Academic Self-Concept 

To assess improvements in students’ math and 
reading self-concept and school engagement, the 
Perception of Ability Scale for Students (PASS) was 
employed. Mean pre- and post-test scores indicate 
that there were significant differences found for 
students’ self-perception of their math ability (from 
9.41 to 10.30; p<.001). Although students improved 
in their perception of their reading ability (from 9.48 
to 9.79), this difference was not statistically signifi- 
cant. There was no difference between students’ pre- 
and post-test scores for student engagement, in fact, 
there was a decrease in student engagement (from 
8.25 to 8.14). 



Program Components 

The BASICS program is designed to target students 
of color living in low-income neighborhoods, who 
are performing below grade level in school. Students 
are recommended to the program by their teachers, 
principals, and parents for help to improve their aca- 
demic and social skills. The program has three major 
goals: (1) improve students’ academic performance 
in reading and math (2) improve the academic self- 
concept of students and (3) enhance the social and 
community skills of students. The following are key 
components of the BASICS program: 

• Voices of Love and Freedom (VLF), a non-profit 
organization that provides literacy, character edu- 
cation, and drug and violence prevention to stu- 
dents. The BASICs program literacy component 
has been fully aligned with several national and 
state language arts learning standards, as well as 
several national standardized assessments. 

• The BASICs program uses Houghton Mifflin’s 
Math Steps curriculum. Math Steps is a compre- 



hensive, research-based curriculum which provides 
a step-by-step approach to developing students’ 
skills and sub-skills, so that all students can be 
successful. 

• Students are given 45 to 60 minutes each day to 
work on homework assignments. 

• An Educational Advisor (EA) is employed at each 
site to develop goals for each student by using an 
assessment from a child’s previous school-year 
teacher, report cards, individual education plans 
(if applicable), and parent notes. Tutors are coached 
by EA’s, who offer them quality and effective 
insight and make sure students’ goals are met. 

• A Site Manager, who may be full-time or part- 
time, depending on the enrollment goals for each 
program, manages his or her individual BASICs 
site. BELL also employs a Program Manager, to 
oversee every program operating in each city. 

• Cultural activities were provided for children to 
learn about their heritage. 



Contributing Factors 

Staff Quality seniors, college and graduate students, teachers and 

The staff is recruited from the community, where the paraprofessional teachers, and other adult community 
school is located. Staff members are high school members. All staff members receive 16 hours of 
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initial training to help them meet the program’s 
objectives. Twenty-four hours of additional training 
throughout the duration of the program is also 
offered. Training includes use of BELL curricula, 
instructional practices, and mentoring. Tutors are 
separately recruited from staff and are used exclu- 
sively for tutoring. They may or may not be paid for 
tutoring. Like staff, tutors are recruited from a pool 
of high school seniors, college and graduate students, 
teachers and paraprofessional teachers, and other 
adult community members. Tutors receive coaching 
from the Educational Advisors. 



Caring Adults 

Staff commit to a one-academic-year relationship 
with students, and work from 3 to 5 days a week. 

National Standard Curriculum 

High quality curriculum that is aligned to state and 

national learning standards 

Small Learning Sessions 

Small teacher to student ratio (i.e., one adult for no 
more than five to seven children) 



Study Methodology 

The study collected pre- and post-data on 208 
program students and compared to this data to 
a national norm group on SDRT-IV and SDMT- 
IV. The test was administered to first through 
fifth graders. The general approach was to 
collect pre and post measures of reading and 
math, using standardized assessment tools, so 
that results were compared to a national norm 
group. Evaluators collected data from all partici- 
pants who enrolled during the program’s first 
and final weeks. 

Evaluation Funding 

The financial support that the BELL organization 
receives is from various funders such as the 
Edna McConnell Clark Foundation and the New 
Profit Inc. The funding that BELL receives provides 
monies for the evaluation of the BASICs program. 

Contact Information 

Research Contact 

Tiffany M. Cooper 

Program Evaluator 

BELL National 

60 Clayton Street 

Boston, MA 02122 

Phone: (617) 282-1567 Ext. 108 

tcooper@bellnational.org 



Implementing Contact 

BELL Boston 

Karimah Ellis 

60 Clayton Street 

Dorchester, MA 02122 

Phone: (617) 282-1567 Ext. 119 

kellies@bellboston.org 

BELL New York 
Tania Pagan 

241 West 139th Street, 3rd floor 
New York, NY 10030 
Phone: (212) 283-9980 Ext. 28 
tpagan@bellnewyork.org 

BELL D.C. 

Karen Baker 
1501 11th Street N.W. 
Washington, DC 20001 
Phone: (202) 462-7206 Ext. 108 
Kbaker@belldc.org 
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Big Brothers Big Sisters 



A Summary of: 

“MAKING A DIFFERENCE: An Impact Study 
of Big Brothers Big Sisters” (1995), by 
Joseph P. Tierney, Jean Baldwin Grossman, with 
Nancy L. Resch 



Program Activity 

Academic assistance 
Cultural enrichment 
Drug & alcohol prevention 
Life skills & training 
/ Mentoring 

Parent & community involvement 
Sports & recreation 



Overview 

Big Brothers Big Sisters of America (BBBS) 
is a nationwide 93-year-old program whose 
autonomously funded local affiliates support 
one-to-one mentoring matches between volunteer 
adults and young people. 

At the time of the study, BBBS maintained 75,000 
active matches between adult volunteers and youth 
as young as five and as old as 1 8. 



Population 

In the sample group of 959 10-to- 16-year- 
olds who applied to BBBS programs in 1992 
and 1 993: over 60% were boys; over 55% 
were members of a minority group (71% of 
whom were African American); 95.6% lived 
with only one parent or grandparent; over 
40% were receiving food stamps and/or 
cash public assistance; almost 55% had 
experienced the divorce, separation or 
death of a parent or guardian; more than 
25% had experienced physical, emotional 
or sexual abuse. 



Key Findings 

Public/Private Ventures (P/PV) conducted baseline 
interviews of applicants to the eight BBBS agencies 
in the study and then randomly assigned the youth to 
treatment and control groups. After 18 months, all 
sample members were reinterviewed. Interviews 
showed that, compared to the control group, youth 
participating in BBBS were estimated to be: 

• 46% less likely to initiate drug use (minority 
Little Brothers and minority Little Sisters were 
70% less likely to initiate drug use) 

• 27% less likely to initiate alcohol use (minority 
Little Sisters were 54% less likely to initiate 
alcohol use) 



• 32% less likely to hit someone 

Participants also reported that they: 

• felt more competent about doing their schoolwork 
(especially minority Little Sisters and white Little 
Brothers) 

• skipped 52% fewer days of school (Little Sisters 
skipped 84% fewer days) 

• skipped 37% fewer classes 

• improved 3% in grade point averages, a surprise 
finding for a non-academic intervention program 
(Little Sisters’ GPAs improved almost 6% and 
minority Little Sisters’ improved 8%) 

• improved the quality of relationships with their 
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parents, primarily due to increased trust 
(especially between white Little Brothers and 
their parents) 

• lied to their parents 37% less 

• improved the quality of relationships with their peers 

Little effect, positive or negative, was shown on 
“self-concept” or on the number of: incidents of 
stolen or damaged property; times youth did “risky” 
things, fought, cheated on a test, used tobacco, were 



sent to the principal’s office or visited a college 
or library; hours per week spent reading and doing 
homework; books read or social and cultural activi- 
ties attended. 

While it is difficult to separate the annual cost of the 
“matched” relationship from other services of the 
BBBS agency, the average cost per “matched” youth 
across 500 agencies is less than $1,000 per year. 



Program Components 

The following features aid the development and 

maintenance of BBBS quality matches: 

• stringent guidelines for screening volunteers by 
professional program staff to eliminate applicants 
who pose a safety risk, are unlikely to keep their 
time commitment, or are unlikely to form positive 
relationships with a young person 

• an orientation for volunteers to explain program 
requirements and rules, with some sites providing 
more extensive training on sexual abuse, develop- 
mental stages of youth, communication and limit- 
setting skills, tips on relationship-building and 
other issues 

• a matching process which takes into account adult 
volunteer, youth and parental preferences; geograph- 
ical proximity of adult volunteer and youth; gender; 



“Big Brothers Big Sisters of America, [is] the 
oldest, best-known, and, arguably, the most sophis- 
ticated mentoring program in the United States'* 
-Public/Private Ventures 

race and religion — most matches are made within 
gender and staff try to make same-race matches 

• supervision to support effective matches, includ- 
ing required monthly telephone contact by agency 
case managers with the adult volunteer and the 
youth and/or parents 

The effectiveness of the matches is likely due to a 
substantial time commitment by both the volunteer 
and youth — both agree to meet two to four times per 
month for at least a year, with a typical meeting 
lasting four hours. 



Contributing Factors 

Caring Relationships 

P/PV: “Our research presents clear and encouraging 
evidence that caring relationships between adults and 
youth can be created and supported by programs, 
and can yield a wide range of tangible benefits” to 
both participants and the larger society. 

Effective Standards and Support Systems 
P/PV: “These findings... do not mean that the bene- 
fits of mentoring occur automatically. The research 
... describes the effects of mentoring in experienced, 
specialized local programs that adhere to well-devel- 
oped quality standards. In our judgment, the stan- 
dards and supports [BBBS] programs employ are 



“Participation in a [BBBS] program reduced 
illegal drug activity and alcohol use, began to 
improve academic performance, behavior and 
attitudes, and improved peer and family relation- 
ships. Yet the [BBBS] approach does not target 
those aspects of life, nor directly address them. 

It simply provides a caring, adult friend! 1 
-Public/Private Ventures 

critical in making the relationships work, and thus in 
generating the strong impacts we have reported. If 
such standards and supports can be duplicated, the 
expansion and replication of mentoring initiatives for 
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early adolescents would appear to be a strong and 
sensible investment, from which at least several 
million youth could benefit.” In contrast to BBBS, 
relatively unstructured mentoring programs were not 
as effective. 

A Developmental Approach 

P/PV: “The findings in this report speak to the effec- 
tiveness of an approach to youth policy that is very 
different from the problem-oriented approach that is 
prevalent in youth programming. This more develop- 



mental approach does not target specific problems, 
but rather interacts flexibly with youth in a support- 
ive manner.” 

Case Managers 

Each match is monitored by BBBS agency case 
managers through required, frequent and direct 
contact with volunteers, participants and their 
parents. Case managers provide guidance and 
support for problems that might arise. 



Study Methodology 

P/PV conducted a random assignment study 
of 959 10-to-16-year-olds who applied to BBBS 
programs in 1992 and 1993. Baseline interviews 
of each applicant were followed by random 
assignment of one-half to the treatment and 
one-half to the control group. 22% of treatment 
group members were never matched. All sample 
members were re-interviewed after 18 months. 

Evaluation Funding 

Lilly Endowment, The Commonwealth Fund, 

Pew Charitable Trusts. 

Geographic Areas 

P/PV chose eight of 500 local accredited BBBS 
agencies for this study. Criteria were a large 
caseload and geographic diversity. The sites 
were Philadelphia, PA; Rochester, NY; 
Minneapolis, MN; Columbus, OH; Wichita, KA; 
Houston and San Antonio, TX; and Phoenix, A Z. 



Contact Information 

Research Organization 

Maxine Sherman, Communications Manager 

Public/Private Ventures 

One Commerce Square 

Philadelphia, PA 19103 

Phone: (215) 557-4400 

Fax: (215) 557-4469 

Implementing Organization 
National Executive Director 
Big Brothers Big Sisters of America 
230 North 13th Street 
Philadelphia, PA 19107-1538 
Phone: (215) 567-7000 
Fax: (215) 567-0394 
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Boys & Girls Clubs of America 



A Summary of: 

"Enhancing the Educational Achievement 
of At-Risk Youth” (2000), Prevention Science, 
1:51-60. By Steven P. Schinke, Kristin C. Cole and 
Stephen R. Poulin, Columbia University School of 
Social Work 


Program Activity 

/ Academic assistance 
/ Cultural enrichment 
/ Drug & alcohol prevention 
Life skills & training 
Mentoring 

/ Parent & community involvement 
/ Sports & recreation 


Overview 

Boys & Girls Clubs of America (B&GCA) was 
founded in 1906 and has more than 2,000 facilities in 
all 50 states, Puerto Rico, U.S. Virgin Islands and U.S. 
military installations abroad. Nearly 400 of these 
programs are in public housing areas. The B&GCA’s 
mission is to form healthy partnerships between 
school-aged children of all backgrounds and con- 
cerned adults. The public housing initiative was 
launched in 1987 under the auspices of the Office 
of Substance Abuse Prevention, U.S. Department of 
Health and Human Services. In 1996, B&GCA piloted 
an after-school educational enhancement program for 
youth in public housing in five cities. This evaluation 
examines the results of the pilot study. 


Population 

Currently B&GCA serves approximately 
three million children, mostly in economically 
disadvantaged areas. The evaluation studied 
992 youth, with an average age of 12.3 
years. 40% were female. Of the participants, 
63.5% were African American, 27.5% were 
Latino, 12% were White and 7.8% other. The 
sample reflected the national population of 
youth who live in publicly subsidized housing 
at the time of the evaluation. 


Key Findings 

In each of the five cities, researchers targeted three 
subgroups of youth to participate in the study: (1) 
youth attending the B&GCA enhancement program 
(“program”); (2) youth from public housing projects 
whose B&GCA did not offer the program (“compari- 
son”); and (3) youth from public housing projects that 
did not have B&GCA (called “control” by researchers). 
Between the pre-test and the 1 8 month follow-up, 
program youth had improved (differences in means 
were statistically significant at the 5% level): 

• Average grade (average grade for program youth 
rose from 78.39 to 83.48, for comparison youth it 
fell from 78.47 to 76.42, and for control youth it 
fell from 75.43 to 71.79). 


• Attendance rates (the mean number of missed days 
in a school year by program youth fell from 6.4 to 
3.7, for comparison youth it rose from 4.85 to 5.85, 
and for control youth it rose from 7.47 to 7.75). 

• Grades in most subject areas (grades were 
rounded to the closest unit to facilitate reading): 

• Mathematics — average grade for program youth 
rose 4 points (from 77 to 82), while falling 3 
points for comparison youth (from 78 to 75) 
and control youth (from 75 to 72). 

• English - average grade for program youth rose 
6 points (from 78 to 84), while falling 1 point 
for comparison youth (from 79 to 78) and 3 
points for control youth (76 to 73). 
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• Writing - average grade for program youth rose 

5 points (from 80 to 85), while falling 1 point 
for comparison youth (from 79 to 78) and 
control youth (from 73 to 72). 

• Science - average grade for program youth rose 

6 points (from 78 to 84), while falling 2 points 
for comparison youth (from 79 to 77) and 4 
points for control youth (from75 to 71). 



• Social Studies - average grade for program youth 
rose 5 points (from 79 to 84), while falling 2 
points for comparison youth (from 78 to 76) 
and 4 points for control youth (from 77 to 73). 



Program Components 

Each week, within the B&GCA facility or in outside 

sessions, the trainers engaged youth in structured 

activities, such as: 

• Four to five hours a week of discussions with 
knowledgeable adults. 

• One to two hours a week of writing. 

• Four to five hours a week of leisure reading. 

• Five to six hours a week of required homework. 

• Two to three hours a week of community services 
(tutoring other children, for instance). 

• Four to five hours a week of educational games, 
such as word and math games. 



Participation is voluntary and, to entice the youth to 
participate, program sites used many incentives, such 
as field trips, school supplies, computer time, special 
privileges, certificates, gold stars and praise. 

Parents are also encouraged to participate with their 
children in the educational activities. Parents and 
youth attended an orientation meeting, after which 
parents were invited to serve as volunteers and to 
attend the cultural events presented by the youth. 

Staff, volunteers and parents attend ongoing training. 



Contributing Factors 

Structured Program 

Some comparison and control sites also offered 
tutoring and homework help, but did not have the 
structure offered by the B&GCA program, did not 
require homework and tutoring, and did not engage 
routinely in educational games to enhance the 
lessons being taught. 



Trained Staff 

Another difference between B&GCA programs and 
the comparison and control sites was the presence of 
a trained staff solely focused on educational 
enhancements. 
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Study Methodology 

This study used both a comparison and a 
“control” group. Because the researchers used 
a nested design, participation in the groups was 
voluntary (not random). Youth were not provided 
with incentives to participate in the study. 
Comparison and control groups mirrored the 
age, gender and ethnic/racial background of 
program youth. Some of the youth in the com- 
parison and control groups received tutoring, but 
did not attend a structured after-school program. 
The attrition rate at the end of the study was 
13.91%, with no significant differences between 
subgroups. Researchers used students’ surveys, 
teacher ratings and school records to collect 
data at the beginning of the program (pre-test), 
six months later (post-test) and 18 months later 
(follow-up). Findings were consistent across all 
measures (this summary presents only school 
data) and differences were statistically significant 
at the 5% level. 

Evaluation Funding 

Carnegie Corporation of New York. 



Geographic Areas 

Public housing projects in Cleveland, OH; 
Edinburgh, TX; New York City, NY; Oakland, CA; 
and Tampa, FL. 

Contact Information 

Research Contact 
Steven Paul Schinke, Professor 
School of Social Work 
Columbia University 
622 West, 1 1 3th Street 
New York, NY 10025 
Phone: (212) 854-8506 
Fax: (212) 854-1570 

Implementing Contact 

Mylo Carbia-Puig 

Director, Prevention Services 

Boys & Girls Clubs of America 

1 230 West Peachtree Street NW 

Atlanta, GA 30309-3447 

Phone: (404) 815-5766 

Fax: (404) 815-5789 

www.bgca.org 

MCPuig@bgca.org 



BEST COPY AVAILABLE 



American Youth Policy Forum 



Finding Fortune in Thirteen Out-of-School Time Programs 



Cap City Kids Program 



A Summary of: 

“An Evaluation Report for the Cap City 
Kids Program: Phase II” (2002) by Dawn 
Anderson-Butcher 



Program Activity 

/ Academic assistance 
/ Cultural enrichment 
/ Drug & alcohol prevention 
Life skills & training 
Mentoring 

/ Parent & community involvement 
/ Sports & recreation 



Overview 

The Cap City Kids (CCK) after-school program was 
initiated by the Mayor’s office in Columbus, Ohio to 
improve youth educational outcomes and access to 
positive youth development programs. In an effort 
to not develop after-school programming that would 
be isolated from the community-at-large, the City’s 
Office of Education coordinated a city-wide forum 
that received the input of more than 180 individuals, 
representing teachers, public school administrators, 
neighborhood representatives, members of the 
Columbus Board of Education, child-care providers, 
child advocates, and other community leaders. What 
emerged from this meeting was a city-wide vision 
for after-school programs focused on providing safe 
places for children, improved academic achievement 
and social development. 

During the months of January to June, 2000, the 
Mayor’s Office of Education piloted the program in 



Population 

There were 181 youth registered for the 
CCK program. Fifty-five percent of these 
youth were female and 45% were male; 87% 
of youth were African American, 12% white 
and 1% Latino. The average income of all 
CCK participants’ families was a little over 
$17,000 and 76% of the children live in 
female-headed households. 



four community recreation centers located in disad- 
vantaged neighborhoods. 1 CCK operates everyday 
after school from 3:00pm - 6:00pm and each site 
enrolls approximately 45 elementary school children 
who live in the neighborhood, immediately sur- 
rounding the centers. Enrollment was based on a 
first-come-first serve basis and there was no charge 
for the program. 



Key Findings 

During this pilot phase of program implementation, 
the only outcome that showed significant differences 
between program students and non-program students 
was school attendance during the third quarter of the 
school year. Although there were no significant dif- 
ferences between both groups of students on teacher 



reported grades in reading, math, and homework 
completion, trend analysis indicated that CCK students 
were increasing their grades in the three areas at a 
higher rate than the non-program students. Related 
findings include: 



'Please note the current evaluation report from the second year (2001-2002) of the CAP City program is now available and the program evaluator may be contacted for 
additional information regarding this report. 
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School attendance 

Youth who participated in CCK had significantly 
lower school absenteeism during the first quarter of 
CCK operations than the matched comparison group. 

Math achievement 

The youth attending CCK had higher math grades 
both before and during their CCK program participa- 
tion than the children in the matched comparison 
group. Both groups increased their math grades 
throughout the school year; however, there were no 
significant differences between the two groups. 

Trends indicate that children attending CCK increased 
these grades at a higher rate than those not attending. 

Reading achievement 

There were no significant differences in reading 



grades and proficiency scores when comparisons 
were made between CCK participants and the com- 
parison groups prior to and during CCK operations. 

Homework completion 

CCK students had higher homework completion 
grades (i.e., a specific category listed on student’s 
reports cards where teachers rate students’ perform- 
ance on out-of-class assignments) both before and 
during their CCK program participation than the 
children in the matched comparison group. Both 
groups increased their homework grades throughout 
the school year; however, there were no significant 
differences between the two groups. Trends indicated, 
however, that children attending CCK increased these 
grades at a higher rate than those not attending. 



Program Components 

The following are program components that are used 
to accomplish the mission of the CCK program: 

• Academic assistance (i.e., homework help, 
computer math) 

• Enrichment activities (i.e., field trips, drama, 
art projects) 



• Prevention units (i.e., anger management, 
alcohol and drug prevention) 

• Recreational activities 

• Family involvement (i.e., parent-staff conferences, 
parenting classes) 

• Nutritious snacks 



Contributing Factors 

Staff Quality 

Site visits to the CCK programs revealed that staff 
are motivated by their work in the after-school 
program and feel supported by the CCK administrator. 
Staff have formed relationships with teachers and were 
building connections with counselors and principals. 



Family strengths and outreach 
Many parents used the CCK program as affordable 
childcare. Parents also served as volunteers and 
worked at CCK sites. CCK programs outreach 
to parents through providing referrals for other 
social services to parents and families, parent action 
meetings, sending children home with food for their 
families, and having parents help plan summer 
program activities. 
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Study Methodology 

Youth attending CCK (n=121) and their matched- 
counterparts (n=119) were compared on various 
academic indicators, including attendance at 
school, proficiency scores, and teacher reported 
reading, math, and homework completion 
grades, four times throughout the school year. 
Since CCK was only in operation during the third 
and fourth grade periods, comparisons could be 
made from before to during CCK program opera- 
tions. The comparison group was pulled from the 
Columbus Public School District’s information 
systems database. 

Evaluation Funding 

Office of Education for the City of Columbus, 
Ohio. 



Contact Information 

Research Contact 

Dawn Anderson-Butcher, Ph.D., LISW 

College of Social Work 

The Ohio State University 

325D Stillman Hall 

1947 College Road 

Columbus, OH 43210 

Phone: (614) 292-8596 

anderson-butcher.1@osu.edu 

Implementing Contact 
Hannah Dillard 
Office of Education 
City of Columbus 
50 W. Gay Street 
Columbus, Ohio 43215 
Phone: (614) 645-0831 
GHDillard@Columbus.gov 
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Juvenile Mentoring Program 



A Summary of: 

Interim Findings, National Evaluation Program Activity 

of the Juvenile Mentoring Program / Academic assistance 

Information Technology International Cultural enrichment 

Drug & alcohol prevention 
Life skills & training 
/ Mentoring 

Parent & community involvement 
Sports & recreation 



Overview 

The Juvenile Mentoring Program (JUMP) was 
implemented in 1995, by the Office of Juvenile 
Justice and Delinquency Prevention (OJJDP) to 
serve youth at-risk for delinquency, gang involvement, 
educational failure and dropping out-of-school. 
OJJDP has competitively awarded funds of up to 
$180,000 - $210,000, on a three-year basis, to over 
250 community-based agencies to begin or continue 
one-to-one mentoring relationships between adult 
volunteers and at-risk youth. One-to-one mentoring 
relationships involved mentors interacting with youth 
an average of once a week, for one hour. To date, 
there are over 260 agencies in 48 states/U.S. territo- 
ries that have received JUMP support. Agencies 
receiving JUMP support must be public or private 
non-profit organizations that are local educational 
agencies (LEAs) or agencies that have developed 
partnerships with LEAs. 



Population 

All JUMP projects must serve at-risk youth. 
There are approximately 17,134 youth 
between the ages of 6 and 18 that partici- 
pate in the JUMP program. JUMP projects 
serve almost an equal amount of males 
(48%) and females (51%). Over half of 
JUMP participants are African Americans 
(51%). Whites (21%) and Latinos (19%) 
account for 40% of JUMP students. Native 
Americans (7%), other (1%), and 
Asian/Pacific Islander (1%) have the small- 
est number of participants in this program. 
The JUMP evaluation gathered data from 
all youth participating in JUMP Projects. 
Findings in this report are based on a 
sample of 451 students between the ages 
of 12 and 14 who completed standardized 
risk screening instruments during their 
participation in JUMP 



Key Findings 

Although the JUMP evaluation includes both a 
process (i.e., site visits and telephone interviews) 
and outcome evaluation (i.e., youth survey), this 
evaluation summary only includes results from the 
outcome evaluation. The outcome evaluation com- 
pared JUMP youth whose mentoring relationships 
had ended with JUMP youth who had just begun 



participation in the project (i.e., generally at the 
beginning of the school year). Both groups of stu- 
dents’ responses were compared on the Problem 
Oriented Screening Instrument for Teenagers 
(POSIT). The POSIT measures 10 psychosocial and 
behavioral domains: 1) substance use/abuse, 2) phys- 
ical health, 3) mental health, 4) family relationships, 
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5) peer relationships, 6) educational status, 7) voca- 
tional status, 8) social skills, 9) leisure/recreation, 
and 10) aggressive behavior/delinquency. Because 
youth taking the survey were between the ages of 12 
and 15, the evaluators did not report information on 
their vocational status. 

• The results showed that post-mentoring youth 
were significantly different than youth just enter- 
ing the JUMP program in the following areas: 

• Higher positive relationships with peers i.e., 
having friends at all, age-appropriate peers, and 
less arguing with friends (mean scores were 3.3 
vs. 2.7) ; 

• Lower aggressive behavior and delinquency, i.e., 
less fighting, gang activity, and use of weapons 
(mean scores were 4.3 vs. 5.0); and 



* Lower feelings of depression and isolation (mean 
scores were 4.7 vs. 5.3). 

Educational status means (i.e., screens primarily for 
disabilities or underachievement due to problems 
with cognitive functioning like concentration, learning 
new tasks, and following directions) were actually 
lower for youth who had long-term mentors (3.8) 
than for youth who did not (3.7). Although these 
mean differences were not statistically significant, 
researchers suggest that perhaps the amount 
of time between test administrations was not long 
enough to show impact on educational achievement. 
Further, having mentors only interact with youth one 
hour, once every week may be too low to impact 
educational status as measured by the POSIT. 



Program Components 



JUMP mentors impact family and peer relationships, 
mental and physical health, and help youth deter 
their involvement in delinquent activities. Some 
JUMP agencies have structured curricula that youth 



and adult volunteer pairs follow, while other agencies 
allow mentor/mentee pairs to participate in activities 
of their own choosing. 



Contributing Factors 

Caring Adults 

The foundation of the JUMP program is centered on 
adult volunteers providing consistent and caring rela- 
tionships to young people that serve as lasting influ- 
ences in deterring their participation in delinquent 
and anti-social activities. All adult volunteers receive 
thorough police background checks. Nearly all vol- 
unteers receive training and some receive ongoing 
support. Because each of the JUMP projects are free 
to structure their project in the way that they feel is 
appropriate, the type of training and ongoing support 
varies widely from one project to the next. 

Project Leaders 

Outside of providing mentors and support to youth, a 
key component for the success of the JUMP program 
is the selection and retention of strong project 
leaders. The strength of project leaders relies on 
their working with youth, volunteers, LEA members, 



and establishing relationships in the community, 
while simultaneously raising the program’s presence 
in the community. Some JUMP projects lack of a 
strong project leader has led to their inability to 
recruit youth and volunteers or to acquire additional 
program resources. 

Collaborative Relationships 

JUMP’S infrastructure is maintained through an array 
of partnerships with LEA’s, community-based organ- 
izations, schools, and project leaders. The most 
important factor in these relationships is that each 
partner’s roles and responsibilities are clearly out- 
lined. The clear demarcating of roles helps each 
group differentiate responsibilities for youth and 
volunteer recruitment, in-kind donations of space, 
equipment or facilities use, and access to informa- 
tion such as school data. 
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Program Sustainability 

Since JUMP programs receive non-renewable 
funding for only three years, it becomes critical 
to programs to obtain additional funding. Some 
programs have failed to identify funding before the 
three-year time period is complete. Failure to identify 
funding has led to a disruption of program services 
and has forced project staff to leave programs early 
for more permanent employment. 



Study Methodology 

This study used a lagged-stage design, whereby 
data from youth completing their mentoring 
matches are compared to youth just entering 
the mentoring project. Youth just entering the 
program complete the POSIT survey. A follow-up 
version of the POSIT is administered at the ter- 
mination of the mentoring relationship. Originally, 
651 youth from 26 JUMP projects completed this 
component of the national evaluation. However, 
because incomplete data were available for 
some of these youth, only 451 POSIT/Follow 
Up POSIT pairs were included in this analysis. 

Evaluation Funding 1 

Office of Juvenile Justice Delinquency 
Prevention. 



Contact Information 

JUMP Program 

Office of Juvenile Justice and Delinquency 
Prevention 

Special Emphasis Division 
810 Seventh Street, NW 
Washington, DC 20531 
Phone: (202) 307-5911 
Fax: (202) 514-6382 
www.ojjdp.ncjrs.org/JUMP 

National Evaluation 

Information Technology International 

10000 Falls Road, Suite 214 

Potomac, MD 20854 

Tel: (301) 765-0060 

Fax: (301) 765-0080 

www. ITI i ncorporated . com 

ITI@ITIincorporated.com 

Technical Assistance Provider 
National Mentoring Center 
101 SW Main Street, Suite 500 
Portland, OR 97204 
Phone: (800) 547-6339 
www. nwrel.org/me ntori ng 



'Evaluators collected data on academic indicators including school grades, attendance and school based discipline. However, because grantees provided insufficient data on 
these measures, evaluators relied on the POSIT scores as an educational indicator. 
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LA’s BEST 



A Summary of: 

“A Decade of Results: The Impact of LA’s 
BEST After School Enrichment Program on 
Subsequent Student Achievement and 
Performance” (2000), by Denise Huang, Barry 
Gribbons, Kyung Sung Kim, Charlotte Lee, and 
Eva L. Baker 



Program Activity 

/ Academic assistance 
/ Cultural enrichment 

Drug & alcohol prevention 
/ Life skills & training 
/ Mentoring 

/ Parent & community involvement 
/ Sports & recreation 



Overview 

LA’s BEST (Better Educated Students for 
Tomorrow) is an after-school program for students 
in grades K-5 in the Los Angeles Unified School 
District (LAUSD). Founded in 1988, LA’s BEST 
provides a safe environment, educational enrichment 
programs, and recreational activities primarily to 
minority students and youth from low-income fami- 
lies. Like many after school programs, LA’s BEST 
runs from the end of the regular school day at 3:00 
pm until 6:00 pm Monday through Friday and there 
is no cost to students’ parents. Unlike other such 
programs, however, LA’s BEST students must regis- 
ter (rather than simply walk-in) to participate in the 
program. Daily enrichment activities include home- 
work assistance, computer activities, drama, sports, 
visual arts, reading and field trips. 



Population 

LA's BEST serves over 18,000 students in 
104 elementary schools throughout the Los 
Angeles metropolitan area. Most of the stu- 
dents who attend LA’s BEST are minorities 
from low-income families. The evaluators 
followed 4,312 LA's BEST participants from 
school year 1993-94 through 1997-98. All of 
the students in the study were in grades 2-5; 
51% were girls; 80% were eligible for free or 
reduced price lunch and 60% were desig- 
nated as having limited English proficiency 
(LEP). The racial/ethnic demographics of the 
students in the study were: 74% Latino, 20% 
African American, 3.5% Asian, and 2.5% 
white. The comparison group of 15,010 
students in LA public schools who did not 
participate in LA’s BEST had higher percent- 
ages of Latinos (79%), Asians (6%), and 
LEP students (67%), and lower percentages 
of students eligible for free or reduced price 
lunch (71%). 



Key Findings 

Evaluators measured the impact of LA’s BEST in 
this study by tracking participant and non-participant 
scores in the areas of reading, mathematics, and lan- 
guage acquisition. Students took the Comprehensive 
Test for Basic Skills (CTBS) and the SAT-9. The fol- 
lowing charts show how a cohort of LA’s BEST stu- 



dents began the 3rd grade scoring below peers in 
the comparison group, but completed the 8th grade 
scoring higher than their peers. For instance, 3rd 
graders began LA’s BEST scoring significantly lower 
on math than their peers, with an average score of 
529 versus an average score of 562 for the compari- 
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son group. By the eighth grade this difference no 
longer exists, LA’s BEST students had an average 
score of 1,051 on reading versus 1,043 for the com- 
parison group. (See below tables for more results.) 

According to the evaluators, one reason for the 
academic achievement for LA’s BEST participants 
was increased school attendance. To compare the 
attendance rate of the 1994/95 cohort of LA’s BEST 
students with their peers’ attendance rate in middle 
school, the evaluators charted these two groups of 
students’ average annual attendance rates in 1998/99. 
It was found that LA’s BEST students attended 
school three more days per year than the comparison 
group. This finding was significant. 



LA’s BEST also helped Latino students became 
more proficient in English than their peers who did 
not attend the after-school program. Evaluators doc- 
umented this finding by charting the proportion of 
students redesignated as “fully proficient” in English 
among LA’s BEST participants and non-participants. 

There were some statistical differences between 
program and non-program students: 

• In sixth grade, 10% of LEP students in LA’s 
BEST versus 6% of LEP non-participants were 
redesignated as fully proficient in English. 

• In eighth grade, 19% of LA’s BEST LEP alumni 
were redesignated as proficient English speakers 
versus 12% of LEP students who had not attended 
LA’s BEST. 



LA’s BEST Achievement Results 1 

Pre-Test Scores: 3rd Grade in 1994/95 

570 - 
560 - 
550 - 
540 - 
530 - 
520 - 
510 - 

Reading Math Language 
■ LA's Best ■ Comparison 



Post-Test Scores: 8th Grade 1998/99 




Reading Math Language 

■ LA's Best ■ Comparison 




Program Components 

There are LA’s BEST programs in 69 schools in 
low-income, Los Angeles neighborhoods. Though 
they vary in size and scope, they share several key 
components: 

• The programs are held in elementary schools from 
the end of the regular school day at 3:00 pm until 
6:00 pm, Monday through Friday. 

• Adults from the school and the surrounding 



community staff LA’s BEST sites. 

• Parents must enroll their children in LA’s BEST 
rather than simply dropping-in sporadically. 

• Student activities include reading, organized 
sports, art, drama, dance, computer work, home- 
work assistance and field trips. 

• Parents are welcome on the field trips and as 
program volunteers. 



'These charts and tables are based on graphs from the LA’s BEST evaluation. The evaluation does not include data sheets, and thus these figures are estimates of the program impact. 
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Contributing Factors 

Extended Learning/Safe Environment 
Parents and students value LA’s BEST because it 
offers a safe place for kids to learn and play after 
school until caregivers come home from work. In 
1993, one quarter of the parents and three quarters 
of the students acknowledged that without LA’s 
BEST, the youth would have had no reliable adult 
supervision after school. 

Parent/Student Commitment 

Evaluators highlighted students’ and parents’ long- 
term commitment to the LA’s BEST after school 
program. The simple factor of enrolling in the 
program rather than dropping-in leads to stronger 
commitment. 

Small Learning Communities 
With no more than 20 students per staff member, 
LA’s BEST provides small learning communities in 
which staff members can give personalized instruc- 
tion and attention to each child. Program staff 
includes management personnel at the Office of the 
Mayor, the Central Office of LAUSD, and more than 
800 site staff 



City-School Partnerships 

LA’s BEST began as an initiative from the Mayor’s 
office that aimed at reducing gang-related activities 
in the city. The partnership between the city govern- 
ment and the schools has continued to be crucial to 
the success of the program. “The city has to be a 
player,” says Carla Sanger, the CEO and President of 
the program. “We can’t leave it to school districts 
alone. There are so many resources that a city can 
make accessible and affordable — theaters, parks, 
field trips, [etc.].” 

School Engagement 

Surveys with parents, teachers and students revealed 
that LA’s BEST improves student attitudes towards 
school. According to teachers, LA’s BEST students 
were more engaged in learning than their peers who 
did not attend the program. 

Program/School Attendance Rates 
Evaluators believed that the combination of students 
regularly participating in the BEST program and 
steadily attending school contributed to BEST’S 
participants’ good school performance. 



Study Methodology 

Researchers followed LA’s Best participants for 
five years to evaluate whether the program influ- 
enced their academic performance. The sample 
consisted of students from all 24 LA’s BEST 
schools in 1994. Within the LA’s BEST schools, 
students who participated in the program were 
identified for the cohort years under study and 
compared to the non-participants in the same 
schools. Cohorts of interest included students 
who were in first, second, third, fourth, and fifth 
grade in 1993-94, 1994-95, 1995-96, and 1996- 
97, and students who were in the first, second, 
third, and fourth grade in 1997-98. Academic 
performance was measured through scores in 
the Comprehensive Test of Basic Skills (CTBS) 
in reading, mathematics, and language arts. In 



1997, the CTBS test was replaced by the 
Stanford-9. Other data included absences in 
middle school, transience, course patterns, and 
changes in LEP status. Results were compared to 
students who did not participate in LA’s BEST. 

The two groups had slightly different ethnic and 
socio-economic composition (see population). 
Participation in the program was divided into three 
levels: high — participants were present more than 
75% days during the five years; medium — partici- 
pants were present from 74% to 26% days 
present; and low — presence 25% of the time or 
less. Linear regression and path analysis were 
used to examine the relationship between inten- 
sity of participation and academic achievement. 
Significance level varies from .000 to .005. 
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Evaluation and Program Funding 

The evaluation was funded by the Bandai 
Foundation, the UCLA Center for the Study 
of Evaluation, and the Office of Education 
Research and Improvement in the U.S. 
Department of Education. LA’s BEST is 
funded by the Los Angeles Mayor’s Office 
and the Los Angeles Unified School District. 

Geographic Areas 

Los Angeles, California 

Contact Information 

Research Contact 

Denise Huang, Project Director 

Center for the Study of Evaluation 

UCLA Graduate School of Education & 

Information Studies 

300 Charles E. Young Drive North 

GSE & IS Bldg. 3rd Floor/Box 951522 

Los Angeles, CA 90095-1522 

Phone: (310) 206-9642 

dhuang@cse.ucla.edu 

Implementing Contact 

Kim Larson, Director of Public Information 

LA’s BEST 

Office of the Mayor 

200 N. Main Street Suite 700 

Los Angeles, CA 90012 

Phone: (213) 847-3681 

Fax: (213) 485-6606 

www.lasbest.org 

klarson@mayor.lacity.org 
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Quantum Opportunities 1 



A Summary of: 

“Evaluation of The Quantum Opportunities 
Program (QOP): Did The Program Work?” 

(1994), by Andrew Hahn, with Tom Leavitt and Paul 
Aaron 



Program Activity 

/ Academic assistance 
/ Cultural enrichment 
Drug & alcohol prevention 
/ Life skills & training 
Mentoring 

/ Parent & community involvement 
Sports & recreation 



Overview 

The Quantum Opportunities Program (QOP) is a year 
round, multi-year, comprehensive service program for 
disadvantaged youth (all from families receiving food 
stamps and public assistance) launched in five com- 
munities in 1989. Twenty-five disadvantaged students 
in each community were randomly selected to enter 
the program beginning in ninth grade and continuing 
through four years of high school. 

QOP is operated by community-based organizations 
in the five communities served (Opportunities 
Industrial Centers in four sites; Learning Enterprise 
in Milwaukee). QOP is a long-term, comprehensive 
program that focuses on three activities to improve 
youth outcomes: (1) 250 hours of educational activities 
(tutoring, homework assistance, computer-assisted 
instruction); (2) 250 hours of development activities 
(life and family skills, planning for the future includ- 
ing postsecondary education and jobs); and (3) 250 



Population 

QOP students were selected randomly from 
families receiving public assistance in each 
of the five project cities. Eighty-six percent 
were ethnic minorities and only 9% lived with 
both parents. 

hours of community service activities aimed at 
improving neighborhood conditions. The QOP 
program provided each site of 25 youth with a caring 
adult that served as both role model and advisor to 
each student for four years. Youth in the program were 
not only expected to form strong caring bonds with 
the adult leader but also with their program peers. The 
QOP program was delivered by community agencies 
that provided services after school on their premises 
and, in some cases, in school settings (where the 
schools provided time and space). 



Key Findings 

Brandeis researchers evaluated four QOP sites. 
Relative to a control group, QOP students: 

• graduated from high school more often 
(63 vs. 42%) 

• dropped out of school less often (23 vs. 50%) 

• went on to postsecondary education more often 
(42 vs. 16%) 

• attended a 4-year college more often (18 vs. 5%) 



attended a 2-year institution more often (19 vs. 9%) 
became teen parents less often (24 vs. 38%) 
were less likely to be arrested (19 vs. 23%) 

more often: took part in a community project in 
the six months following QOP (21 vs. 12%); were 
volunteer tutors, counselors or mentors, (28 vs. 
8%) and gave time to non-profit, charitable, 
school or community groups (41 vs. 1 1%, only 
statistically significant at the Philadelphia site) 



O 

ERIC 
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The effects of QOP increased over time, as measured 
at the end of each high school year. After the first 
year, there were no significant differences seen 
between the QOP and control groups in the 1 1 aca- 
demic and functional skill areas measured. After two 
years, scores of QOP participants were higher in all 
1 1 areas, and the difference was statistically signifi- 
cant in five areas. By the time QOP students and 
control sample were leaving high school in 1993, 
QOP student group scores in all 1 1 areas were much 
higher than control student scores, and the differ- 
ences were statistically significant in every area. 

There was wide variation among the program sites. 
One of the five original sites, Milwaukee, was 
dropped from the evaluation after problems with 
implementation and follow-up. Of the remaining 
four, Philadelphia had the most significant out- 



comes. For example, the rate of four-year college 
attendance was nearly three times higher than the 
rate in San Antonio, five times higher than 
Oklahoma City, and eight times higher than Saginaw. 
Researchers noted that at the Philadelphia site, staff 
developed and maintained strong bonds with the 
QOP students, and were able to forge a cohesive 
group identity. 

The Ford Foundation forward funded QOP at $1.3 
million for four years. The evaluation’s cost/benefit 
analysis showed that QOP cost $10,600 per partici- 
pant over the four year period and that $3.68 was 
gained for every dollar spent if QOP college students 
earned a degree. Even if only one-third of QOP 
college students ultimately received degrees, the 
benefit-cost ratio was $3.04 for every dollar spent. 



Program Components 

QOP also features financial incentives for partici- 
pants and staff. Students receive small stipends for 
participating in program activities (starting at $1 per 
program hour, and rising to $1.33) and bonuses for 
completing activities ($100 for every 100 program 
hours). They also receive a matching amount in an 
account that could be used only for post-program 
activities, such as college and training. 



“In contrast to most youth programs in the 
‘ add-on ' or ‘ second-chance ’ tradition, QOP was 
designed to encourage long-term involvement 
through an array of services. Meaningful 
relationships with adults would be encouraged 
without fear of having bonds abruptly severed 
when the programs ended.” 

-Brandeis University 



Contributing Factors 

Caring Adults 

Brandeis: “If young people are connected with 
caring adults for sustained periods of time, year- 
round, positive results do emerge.” Program adminis- 
trators and staff, as well as teachers and mentors, 
took an active interest in the welfare of the QOP 
students, encouraging them, visiting them, following 
up and doing everything they could to keep them in 
the program. “Once in QOP, always in QOP” was the 
unofficial motto, and most program counselors took 
it to heart. 



“Simply put, when a quantum opportunity was 
offered, young people from public assistance 
backgrounds — African American males, females, 
whites, Asians, others — took it! They joined the 
programs and many stayed with the programs 
or the staff associated with the initiatives, for 
long periods” 

-Brandeis University 
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Sense of Community 

The project sites are small, with only 25 students in 
each. Students are able to bond with each other and 
with adults in the program, particularly at the 
Philadelphia site. 

Multiple Services Encompassing All Aspects of 
Youths 9 Lives 

The QOP program was designed to address the many 
challenges and obstacles that disadvantaged youth 
face. QOP focuses on developing basic skills (aca- 
demic and functional) for future success, strengthening 
life and social skills to make better choices and 
operate more effectively with families and peers, 
broadening horizons through cultural trips and other 
experiences, and taking pride in the community 
through active service. 

Quality Staff 

Results from the most effective project site — 
Philadelphia-show what can be accomplished with a 
dedicated, quality staff. Brandeis: “The differences, 
for example, between San Antonio and Philadelphia 



cannot be attributed to the neighborhood setting, the 
characteristics of participants, or to the program 
model. What distinguishes these sites is the degree 
of buy-in from the host organizations and the com- 
mitment of staff at all levels.” 

Financial Incentives as Part of a Comprehensive Program 
While financial incentives are important to some stu- 
dents, and help with family expenses, it appears that 
they are not the decisive factor in QOP participation. 
When they are part of a comprehensive, well-devel- 
oped program, financial incentives can be effective 
in maintaining student interest in and attendance at 
program events. However, they do not appear to 
operate effectively in the absence of a strong program 
featuring much personal contact with staff. 

Financial Resources 

The Ford Foundation funded the QOP program 
upfront, making it possible to plan for and deliver 
a host of services over an extended period of time. 
Both staff and students knew the resources were 
there to carry through on their commitment. 



Study Methodology 

In 1989, program designers randomly assigned 
50 disadvantaged students in each of the five 
sites to either a program or a control group. 
Researchers compared the progress of the two 
groups with periodic questionnaires and basic 
skills tests. 

Evaluation Funding 

Ford Foundation. 



Contact Information 

Research Organization 
Andrew Hahn 

Center for Human Resources 

Heller Graduate School, Brandeis University 

Waltham, MA 02254-9110 

Phone: (617) 736-3774 

Fax: (617) 736-3851 

Implementing Organization 
C. Benjamin Lattimore 
Opportunities Industrialization Centers of 
America, Inc. 

1415 Broad Street 
Philadelphia, PA 19122 
Phone: (215) 236-4500, Ext. 251 
Fax: (215) 236-7480 



1 For the evaluation of the replication of QOP, please see: Maxfield, M., Schirm, A., & Rodriguez-Planas, N. (2002). The Quantum Opportunity Program Demonstration: 
Implementation and Short-Term Impacts. Washington, DC: U.S. Department of Labor. 
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Sacramento START 



A Summary of: 




“Supporting Student Achievement: 
Evaluation Report for 2000/2001” 

(2001) by Minicucci Associates 


Program Activity 

/ Academic assistance 

✓ Cultural enrichment 
Drug & alcohol prevention 

/ Life skills & training 
Mentoring 

✓ Parent & community involvement 
/ Sports & recreation 


Overview 




Sacramento START (Students Today Achieving 
Results for Tomorrow) is a regional after-school 
program that is operated by the city of Sacramento 
in collaboration with the county of Sacramento and 
five area school districts. The schools served by 
START programs are located in areas with wide- 
spread poverty, unemployment, numerous single 
and teen parents, low education levels, high dropout 
rates, crime and violence. The primary focus of 
START programs is to provide academic support 
to students who are struggling in school. 


Population 

The evaluation study included 748 students. 
54% of the students were females. Of the 
748 students, 30% were third graders, 33% 
fourth graders, 24% fifth graders, and 13% 
sixth graders. Twenty-nine percent of the 
students were Latino, 27% African Americans, 
23% Asian, 15% white, and the remaining 
1 % “Other.” All participants were residents of 
the Sacramento metropolitan area. 


In 1995, the START program was founded and 
placed in five school districts and 1 8 elementary 
schools. Currently, the START program serves 3,820 
elementary school students. The city of Sacramento 
serves as the fiscal agent for the program, and also 


employs and trains staff and prepares reports to 
funders. Elementary school sites provide space for 
the program and school personnel collaborate on 
curriculum planning and support the program 
financially with matching funds. 


Key Findings 



The evaluator used SAT-9 standardized test scores 
from the different START schools and districts, 
reporting the data in Normal Curve Equivalent (NCE) 
scores, based on national test performance. The state 
of California asks districts to examine growth in 
learning for students in the bottom 25th percentile 
or rank 1 , therefore, these findings are also reported. 



Academic Achievement 

Pre/post data on SAT-9 scores from six START 

school districts showed: 

• START students significantly improved 3 NCE 
points on their math SAT-9 scores; 

• START students significantly dropped 1 NCE 
point on their reading SAT-9 scores 1 , there was no 
change in reading scores for English Language 
Learners; 
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• In math, English Language Learners gained 4 
NCE points compared to English speakers who 
gained 2 NCE points; 

• For reading, 37% of START students were in the 
bottom quartile, 20% of those students moved to 
a higher quartile; and 

• For math, 30% were in the bottom quartile, and 
42% of those students moved to a higher quartile. 

School Attendance 

Students who attended START programs in the pre- 
vious year improved their school attendance the fol- 
lowing year from 5.5 days absent to 5.4 days absent, 
a reduction of 0.1 days absent. 



Students deemed “problem attenders” (i.e., missing 
more than 10 days of school) in the previous year, 
improved their attendance the following year, drop- 
ping from 14.8 days absent to 10.1 days absent, a 
reduction of 4.7 days absent 

A study of a small sample of 227 returning START 
students showed improved attendance in the first 
year they were in START (7.3 days absent), second 
START year (5 days absent), and third START year 
(5.1 days absent), a total reduction of 4.7 days absent 
from school. 



Program Components 

Academic Support 

Classroom teachers were instrumental in providing 
academic support to START programs and students. 
Teachers served as literacy coaches who worked 
closely with START staff to link classroom literacy 
programs to after-school START activities. Teachers 
also helped staff establish learning centers, explain- 
ing classroom techniques, instructional activities, 
and providing recommendations to program leaders 
on cluster activities. 

Enrich ment A divides 

START programs provided enrichment activities that 
included performing arts, recreation, field trips, and 
curriculum connected to the local newspaper. All 
enrichment activities contained a learning component. 

Staff Training 

START staff are involved in a number of trainings 
that reflect the program implementation process and 
principles. Trainings are focused on core topics and 



cluster training. Core trainings are required for all 
site directors and program leaders and consist of 
workshops in classroom/behavior management, 
enrichment, homework/tutoring, literacy, attendance, 
and lesson plans. The START training program is 
a key strategy for aligning after-school program 
activities with the schools’ curriculum. During the 
summer of 2000, START staff participated in more 
than 200 hours of training. 

Building Community Capacity 

One of the primary objectives of the START program 
is to build the capacity of the community surround- 
ing each site by providing employment and training 
for neighborhood residents, parents and school 
instructional aids. 

Student/Staff Ratio 

The majority of START sites had a student-to-staff 
ratio no greater than 20: 1 . START directors are 
trying to lower this ratio. 



Contributing Factors 

School/Program Collaboration 
Communication and collaboration between START 
directors and school administrators was crucial to 
the success of the program. START had to work with 
schools, especially in aligning the academic training of 
staff and the learning goals of students in the program. 



Extended Learning/Sense of Safety 
By providing a safe and fun-learning environment 
after school, the START program offered an alterna- 
tive avenue of academic enrichment for minority and 
low-income students. 
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Community Involvement 

START consciously worked to involve members of 
the community in its after-school program, hiring 
nearly three-quarters of its staff from neighborhoods 
surrounding the elementary schools where the 
program was held. 



Study Methodology 

The evaluation team selected a pool of 1200 stu- 
dents in grades three to six who participated in 
START for 30 days during the months of 
February and March. Of these students, only 
748 met the evaluation criteria of having com- 
plete achievement data for 1999/2000 and 
2000/2001 . The evaluators also included 
achievement data on 227 START students 
returning to the program for the second year, 
findings on 707 students with complete informa- 
tion on school attendance who were enrolled for 
a full year in the program year and the previous 
year and the 186 returning START students with 
complete attendance information for three years. 
The evaluators collected data through focus 
groups, surveys, and interviews. 

Evaluation Funding 

The City of Sacramento funded the evaluation 



Geographic Areas 

Sacramento, CA 

Contact Information 

Research Contact 
Minicucci Associates 
1608 I Street, Suite 100 
Sacramento, California 95814 
Phone: (916) 442-4720 
minicucci@jps.net 

Implementing Contact 

Andee Press-Dawson 

6005 Folsom Blvd 

Sacramento, CA 95819 

Phone: (916) 277-6022 

Fax: (916) 277-1214 

APDawson@cityofsacramento.org 
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1 California state laws requires that schools provide information on a minimum of 25 students for each grade level participating in the program. 
See also: Minicucci Associates (November 2002). Sacramento START Evaluation Report 2001/2002. Sacramento, CA: Author. 
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Youth Education for Tomorrow 



A Summary of: 

“Mustering the Armies of Compassion in Program Activity 

Philadelphia” (2002) by Bill Hangley, Jr., / Academic assistance 

& Wendy S. McClanahan Cultural enrichment 

Drug & alcohol prevention 
Life skills & training 
Mentoring 

Parent & community involvement 
Sports & recreation 



Overview 

In 2000, Public Private Ventures (P/PV) was awarded 
the Community-Serving Ministries grant, by the Pew 
Charitable Trust, to explore the capacity and potential 
of faith-based institutions in Philadelphia to provide: 
(1) literacy, (2) mentoring, and (3) child-care services. 
Literacy was the first area of focus for P/PV To 
examine the capacity of religious institutions to 
improve the literacy levels of youth, P/PV established 
the Youth Education for Tomorrow (YET) center 
model. The YET model was established with the 
belief that any site of faith, despite size or financial 
resources, could successfully teach children to read. 
In this model, sites provided the space, the children, 
and the volunteers while P/PV provided the funding, 
pedagogy and training for all involved staff. With the 
YET model as a framework, sites instructed young 
people in making the connection between reading, 
speaking, writing, and the events of the world 
around them. P/PV cautioned that the YET model 
was not a clinical program, nor was it intended to 
diagnose or serve children with severe reading 
deficiencies. The program was also not designed 
for those who already read at grade level, although 
they may have benefited from the program. P/PV 



Population 

The YET program served 885 youth. Eighty 
percent of enrolled youth were African 
American, 7% Latino, and 13% were white. 
Fifty-seven percent of program youth were 
males and 43% were females. Sixty percent 
were in grades 1 to 5, 10% were in grades 
6 to 8, and 21% were in high school. More 
youth were served that were living below or 
at the poverty level (i.e., 75% reduced free 
or reduced price school lunch). Students in 
the YET program averaged grades of B 
minus to C and 2% of YET students were 
in special education. 

required sites to restrict the program to children who 
read three years or less below grade level. The 21 
sites that participated in this program were from 
a variety of religious affiliations, e.g., Baptist, 
Catholic, Mennonite, Presbyterian, and Methodist 
denominations. Sites were located in schools, 
community-based organizations, and several 
area churches. 



Key Findings 

Literacy Skills Improvement 
Elementary Students 

The average elementary student who attended the 
YET program was a third grader who read a little 



better than a first-grader (1.7). After 90 days of 
attendance in the YET program, students improved 
their reading ability to read 0.5 grade level below 
their grade level, for an improvement of 1.2 grade 
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levels. This means that the average third grade 
student who arrived in the YET program reading 
a little better than a student in the first grade, after 
90 days in the program, was reading well into the 
second grade. 



Middle and High School Students 

The older children (grades 6 to 12), according to a 
pre-test using the Informal Reading Inventory, were 
reading 3.9 below grade level. When tested after 90 
days, students were reading 2.2. below grade level, 
for an improvement of 1.6 grade levels. These stu- 
dents reduced their deficiency by an average of 58%, 



Program Components 

• The 21 YET sites contain approximately 30 youth. 
The reading ability of elementary children who 
came to the program was 1 .7 years below grade 
level. The reading ability of older students who 
came to the program was 3.8 years below grade 
level. Although the goal of the YET program was 
to target students below grade level and bring them 
up to grade level, sites had moral difficulty exclud- 
ing young people with proficient reading ability. 

• YET center classes are held after school for 90 
minutes, four days a week, throughout the school 
year. During this time period, students and a 
teacher (ratio) engage in four basic activities: 



(1) oral language vocabulary activity i.e., talking 
about words and ideas in the context of current 
events, holidays, issues in other things of interest 
to students, (2) teacher reading aloud to students 
so students hear what good reading sounds like, 
(3) student reading, and (4) writing. 

• Sites are required to post a daily schedule of 
classroom activities, displays of students’ work 
and a “word wall” where students could view 
high-frequency words. Sites are also asked to 
make sure students had library cards and access 
to books in class. 



Contributing Factors 

Staff Quality 

Teachers are professionally qualified and approved 
by P/PV 

Student Attendance 

The more a student attended the YET program, the 
greater gains they made in reading improvement. 
Students who attended 100 days or more at program 
sites, averaged almost two grade levels (1.9) of 
reading gains, whereas students who attended less 
than 100 days reading gains only improved a little 
over one grade level (1.1). The effect of program 
attendance on reading improvement remained critical 
even after such individual characteristics as race, 
gender, age, and low-income status were considered. 



Program Implementation 
The more effective sites implemented the YET 
program model, and reading scores for students 
improved. P/PV rated YET centers adherence to the 
program model based on program site visits. Each 
site was assigned a score from 1 (poor) to 5 (out- 
standing) in each of four categories (oral language, 
writing, reading, and overall consistency). YET 
centers that scored higher on P/PV’s rating scale had 
higher performing students than centers who scored 
lower on P/PV’s rating measure. 
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Study Methodology 

Children were tested for their reading levels 
at their entry into the program, midway in the 
program and at the end of their first year in the 
program. Students were assessed with the 
Informal Reading Inventory that provided 
teachers with students reading levels. 

Evaluation Funding 

The Pew Charitable Trust. 



Contact Information 

Bill Hangley Jr., 

Public Private Ventures 
2000 Market Street 
Suite 600 

Philadelphia, PA 19103 
Phone: (215) 557-4400 
Fax: (215) 557-4469 

Wendy S. McClanahan 
Public Private Ventures 
2000 Market Street 
Suite 600 

Philadelphia, PA 19103 
Phone: (215) 557-4400 
Fax: (215) 557-4469 
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Appendix I 



The evaluations included in Some Things DO 
Make a Difference for Youth: A Compendium of 
Evaluations of Youth Programs and Practices (1997); 
MORE Things That DO Make a Difference for Youth 
(1999); Raising Academic Achievement (2000); 
Raising Minority Academic Achievement (2001); 
and No More Islands: Family Involvement in 27 
School and Youth Programs (2003) were selected 
through a multiphase process: 

A. Collection of evaluations : The searches involved 

(1) reviews of national databases, such as the 
Educational Resources Information Center 
(ERIC), Sociological Abstracts and the National 
Criminal Justice Reference Service (NCJRS); 

(2) Internet searches; (3) direct contact with 
program coordinators, policymakers, funding 
officers and researchers; (4) distribution of flyers 
requesting evaluations during forums, confer- 
ences and similar events; and (5) a request for 
evaluations posted at the AYPF web site 

(http : // www. aypf . org) , 



B. Initial selection : The evaluations collected were 
then reviewed for (1) program characteristics — 
programs and practices had to target school-aged 
children and older youth; (2) research quality — 
evaluation sample, design and methodology had 
to follow accepted research standards; and (3) 
program results/outcomes — the evaluations had 
to include quantitative data indicating the initia- 
tive resulted in positive effects on participants, 
such as improved academic achievement, 
increased graduation rates, decreased rate of 
risky behaviors, and others. 

C. Internal review : The evaluations that met the 
criteria above were summarized and reviewed by 
an internal committee. Summaries approved in 
this initial review were then sent to evaluators 
and program staff members to clarify questions, 
provide more recent data when available, and 
ensure each summary’s accuracy. 

D. External review : An external reviewer read all 
the selected summaries from the reports to 
assess once more the quality of the original 
research and the summaries. 



American Youth Policy Forum 



• » 



Finding Fortune in Thirteen Out-of-School Time Programs 

References 



Alberta Mental Health Board (2001). Take time you can make a difference to a child's mental health 
[Brochure]. Retrieved February 9, 2003, from http://www.amhb.ab.ca/chmh/TakeTimeBrochure2 1 .pdf 

Center for Human Resources (1995). Quantum Opportunities Program: A brief on the QOP pilot program. 
Waltham, Massachusetts: Center for Human Resources. 

Erikson, E.H. (1968). Identity: Youth and Crisis. New York: W.W. Norton & Company. 

Gambone, M.A., & Arbreton, A.J. (1997). Safe havens: The contributions of youth organizations to healthy 
adolescent development. Philadelphia: Public Private Ventures. 

Gottfredson, D.C., Gottfredson, G.D., & Weisman, S.A. (2001). The timing of delinquent behavior and its 
implications for after-school programs. Criminology & Public Policy , 1(1), 61-86. 

Hart, K. (2000, Fall). An interview with Michelle Seligson. The Evaluation Exchange , 6(1). Retrieved 
February 8, 2003 from http://www,gse,harvard.edu/hfrp/eval/issuel5/html/questions.html 

Miller, PH. (1993). Theories of developmental psychology. New York: W.H. Freeman & Company. 

Olsen, D. (2000). 12-hour school days?: Why government should leave afterschool arrangements to parents. 
Retrieved February 8, 2003, from http://www.cato.org/pubs/pas/pa-372es.html 

Posner, J.K., & Vandell, D.L. (1994). Low-income children’s after-school care: Are there beneficial effects 
of after-school programs? Child Development , 65, 440-456. 

Rossi, P.H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5 th ed.). Newbury Park, 
California: Sage Publications. 

Scales, P, & Leffert, N. (1999). Developmental assets: A synthesis of the scientific research on adolescent 
development. Minnesota, MN: Search Institute. 

Schinke, S. R, Orlandi, M. A., & Cole, K. C. (1992). Boys and girls clubs in public housing developments: 
Prevention services for youth at risk. Journal of Community Psychology , 28, 118-128. 



BEST COPY AVAILABLE 




American Youth Policy Forum 



69 



AMERICAN YOUTH POLICY FORUM 




■ P O L I C Y ■ 



1 836 Jefferson Place NW 
Washington, DC 20036 
www.aypf.org 



ISBN: 1-887031-81-2 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 



X 



This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 




EFF-089 (1/2003) 



